University of Arkansas Office for Education Policy

Archive for the ‘The View from the OEP’ Category

Education Committees Discuss Funding Ahead of Fiscal Session

In The View from the OEP on February 10, 2016 at 11:49 am

capital-pic

 

Based on discussions in this week’s meetings of the Education Caucus and the House and Senate education committees, we expect that use of NSL state funds will be the main education-related topic at the 2016 Fiscal Session scheduled to begin April 13. Transportation funding may get some attention if legislators come up with an alternative approach to allocating funding more equitably among the wide-ranging needs of districts.

 NSL State Funds

The Education Caucus discussion focused on NSL state funds. Legislators are frustrated that districts’ use of these general revenue dollars aimed at closing the achievement gap between poor students and their more affluent peers has not resulted in the expected improvement. About $210 million of categorical NSL state funds were given to school districts and charter schools in 2014-15, up from $160 million in 2008-09, as the proportion of students qualifying for free and reduced price lunches has increased. (See BLR’s detailed research report.) Lawmakers are questioning the effectiveness of the 28 or so eligible expenditures and considering whether the list should be narrowed to programs with the most support in research. At the Education Caucus meeting, advocacy organizations further explained their ideas that were part of larger reports presented earlier this year (OEP blog post January 13).

  • The Walton Family Foundation (WFF) recommended designating a portion of the NSL funds as performance-based, phasing in to 20% performance-based by 2021, saying that eliminating options runs counter to the state’s emphasis on innovation.
  • Arkansas Advocates for Children and Families (AACF) recommended limiting the uses of NSL state funds to those with the strongest evidence base, such as pre-kindergarten, after-school, and summer programs.
  • Arkansas Association of Educational Administrators (AAEA) liked the idea of incentives, but Executive Director Richard Abernathy disagreed with “cutting 20% from funding and then calling it an incentive.” He said AAEA supports using funds for the most effective programs but urged lawmakers to “leave alone” the schools where programs are working. Abernathy also recommended “smoothing the funding cliffs” and keeping the funds as a separate category rather than combining with something else.
  • Arkansas School Boards Association concurred with the AAEA view that incentives are helpful but not if taken from current funding. Saying that some programs reinforce each other, Executive Director Tony Prothro cautioned that eliminating one program may have the unintended consequence of making other programs less effective.

Senator Alan Clark (R-Lonsdale) asked if policymakers “should look at poverty in a different way,” saying that the federal poverty standard may not accurately reflect the situations of Arkansas students when it comes to distributing NSL state funds.

OEP will keep you posted as legislators continue to grapple with NSL, transportation, and other Fiscal Session issues.

 

 

Commit. Serve. Teach.

In The View from the OEP on February 3, 2016 at 11:26 am

Teach For America (TFA) was big news in Arkansas last week, and today we wanted to highlight a similar Arkansas-specific program that places high quality teachers in high-need schools in Arkansas. The deadline to apply for this outstanding program is March 6th, 2016.

The news about TFA came when Gov. Asa Hutchinson announced funding to support 150 new TFA teachers to be assigned to school districts in south and east Arkansas.  In addition, members of the Little Rock business community were donating an additional $3 million in private funds to hire some 65 new TFA teachers in the Little Rock School District specifically over the next three years.

Here at OEP we have had the pleasure of working with many TFA teachers, and have found them to be smart, hard-working, dedicated professionals who care about educating their students. We are excited about students being taught by folks like these.

The fact is that teachers matter to kids. We support high quality teachers in classrooms, and have found great teachers can originate from sources other than traditional teacher training programs.

ATC Logo

The Arkansas Teacher Corps provides a ‘home-grown’ path to making a difference for Arkansas students.  There are several benefits to becoming an ATC Fellow (what they call their teachers) that prospective difference-makers should consider:

  • More Commitment: ATC Fellows commit for three years. Longer than the TFA commitment, ATC Fellows provide more stability to students and schools.
  • More Local: Arkansas Teacher Corps provides a ‘home-grown’ path to teaching in Arkansas, for Arkansans. Training takes place in Arkansas, addresses the needs of Arkansas students, and the vast majority of ATC Fellows are from Arkansas.
  • More Opportunities: ATC Fellows are placed in low-income or rural districts throughout Arkansas, because there are students in all regions who can benefit from a high quality teacher.
  • More Support: There are currently 43 ATC Fellows teaching in Arkansas. TFA’s current corps is over 8,600, with 110 serving in Arkansas. ATC’s personalized professional support, close relationships with school districts, and a strings-free Fellow stipend of $15,000 over the three years in addition to a regular teaching salary, ensures that ATC teachers get the support they need to be effective educators for students.

Arkansas students are waiting for great teachers!  Every year, ATC has many more requests for teachers than they can fill. Like Gov. Hutchinson noted about TFA: once district leaders work with ATC, they continue to request more teachers from the organization.  Districts need great teachers, and you can help!

ATC is looking for service-minded individuals from all different backgrounds. Please pass this information along to people you know who want to make a difference in the lives of Arkansas students.  Check out http://arkansasteachercorps.org/ to learn more.

 

No *s!

In The View from the OEP on January 27, 2016 at 12:07 pm

asterisk

When reviewing the preliminary PARCC assessment results at the December Education Caucus, Senator Clark told ADE staff that he wanted to see a data report “without asterisks”.  Although we didn’t see the document Senator Clark was referring to, the asterisks were presumably being used to indicate that although several states administered the same PARCC assessments last year, results between Arkansas and some other states were not directly comparable.  As noted in our recent blog on statewide PARCC results, here at OEP we found that even before considering the varied poverty levels of the states, it was appropriate to refrain from making direct comparisons in some cases.

In the 2014-15 school year, 5 million students in 11 states and the District of Columbia took the PARCC assessments in grades 3-11,  but not all participating states had students in all grades taking the test.  Education is a state responsibility, and states were free to set individual expectations for PARCC testing.  In Massachusetts for example, districts were allowed to choose if they wanted students to complete the PARCC tests or the MCAS, the state test used previously.  In Ohio, 3rd graders didn’t take the English Language Arts portion of the PARCC.  In Arkansas, the Algebra II and 11th grade literacy tests were optional for districts. PARCC testing expectations for high school students were the most varied across the states, and therefore the most inappropriate to compare.  Here at OEP we feel you can compare performance in the earlier grades, but still must consider the possible impact of varying degrees of poverty across the states.

So, for ONCE we were taking the same annual test as 10 other states, and we STILL CAN’T COMPARE student performance?  *!  We understand, Senator, all of this is super frustrating.  You just want to know how Arkansas students are doing.

It’s frustrating for Arkansas parents too, because… how do we know if our kids are on track for success?

Here at OEP we wanted to share some suggestions to help parents and policymakers appropriately interpret various sources of information about student performance.

PARCC scores*

Only 1% of Arkansas students exceeded expectations in mathematics, and only 4% exceeded expectations in English Language Arts. We aren’t Lake Wobegon, where all the children are above average, but what a change! Proficiency percentages dropped from over 70% last year to mid 30% with the change in assessment this year.  The expectations of student performance are much higher on PARCC than on the prior assessments, and here at the OEP we feel they are more consistent with expectations on other key assessments such as the NAEP and ACT, and more reflective of the skills students will need to be successful in college and careers.

From a parent perspective, however, interpreting the PARCC scores can be confusing.  There is a lot of information out there about understanding the score, but we still aren’t quite sure what to make of it as parents.  The average score is provided for a student’s district, state and the PARCC states overall, but there is no percentile rank provided, so the best parents can tell about a student’s overall performance is that it was better or worse than average. 

We all know of course, that one score isn’t the whole picture of a student.  The PARCC site notes, “This information, along with grades, teacher feedback and scores on other tests, will help give a more complete picture of how well your child is performing academically.”

Here at the OEP, we always suggest using multiple sources of data- but get ready for more asterisks!

Grades*

Grades are SOO subjective. If a student gets a ‘B’ in a 5th grade class in School A, is there any reason to think that that same student would get a ‘B’ in a 5th grade class in School B?  Not really!  Teachers assign grades based on their own criteria.  This can include completing homework, class participation, attendance, and performance on teacher-created projects and assessments.  Sometimes grades are a better measure of compliance than student’s knowledge about a subject.  According to the annual grade inflation report, in 2014 over 7% of students in Arkansas received a grade of ‘A’ or ‘B’ in their high school math classes and yet were not proficient on the state-mandated End of Course test.  While this may not seem too concerning (we all suffered through high school math), 12% of high schools in the state were found to have widespread ‘grade inflation’.  In these 39 schools more than 1 in 5 students didn’t pass the state test even though they were awarded an ‘A’ or ‘B’ in the relevant class.  These students were getting a grade that indicated they were on track for success, but had not mastered the content.  Student grades should definitely have an asterisk because they are NOT comparable between teachers, schools, districts or states.

Teacher Feedback*

Here at the OEP we value feedback from teachers about student performance.  Teachers spend time helping students learn every day, and we respect their expertise and knowledge about kids. Experienced teachers have seen a lot of students pass through, and can have an informative perspective on how a student compares with the ‘typical’ student they see.  A limitation to this can be that ‘typical’ is based on what he or she has experienced.  Being a 5th grade classroom teacher in Arkansas for 7 years does not necessarily make you an expert on how a student is progressing toward preparation for college and careers. You might have a different frame of reference if you were teaching in a different school or another state.  Teacher feedback should have an asterisk because it is one perspective that is NOT comparable between teachers, schools, districts or states.

Other Test Data

Hooray!  This is one piece of information that may not need an asterisk if the right type of assessment is used.  NWEA MAP assessments, used by many districts throughout Arkansas, can provide valid and reliable information on student performance and growth compared to a nationally representative sample.   MAP data shows how a student’s performance compares to other students across the country, and is available for students in grades K-12.  The individualized growth targets can keep track of how any student, regardless of performance level, is keeping pace, catching up, or falling behind his or her academic peers.

And now using this COOL NEW TOOL, students and parents can explore what colleges and universities students are on track to attend just by inputting MAP scores. AND- you can use this tool for students as early as fifth grade!

According to NWEA:”The College Explorer Tool uses correlations between MAP scores and college readiness benchmarks for the ACT to pinpoint the colleges and universities for which a students’ forthcoming scores would likely be near the median admissions scores. Additionally, the tool provides a quantitative profile of each institution using data from the U.S. Department of Education’s College Scorecard, which includes valuable information on cost of attendance and the average annual cost borne by families at different income levels. This crucial information shows students and their families how much they would likely need to borrow in order to complete their education at a given college, as well as graduates’ typical earnings.”

Screen Shot 2016-01-27 at 9.57.34 AM

 

 

 

 

 

Fifth grade might seem young to be thinking about post-secondary options, but the incredible thing about having this information available so early in the academic process is that it gives kids a heads up in time to CHANGE THE TRAJECTORY.  If a student wants to go to UofA, or CalTech, or University of North Dakota, he or she can see if they are on track to meet the typical ACT/SAT score acceptance criteria. If the student is falling short there is time to close the gap!  Of course, there is way more to being successful in college and career than test scores, BUT such early information may help kids expand their horizons and reach their dreams.

Because it is a stable and nationally representative assessment, NWEA MAP scores can also be used to nominate students for early talent identification programs, and special school-level growth norms can help determine if programs are effective. This computer-adaptive assessment with strong psychometric properties and a large national sample can help students, parents, teachers and school leaders look at some information with no asterisks.

Comparing Responsibly

In a quest for improving education for students, we are always trying to find what works-  did Program A work better than Program B? It’s complicated because a lot of things can impact educational outcomes, and there are lots of different measures of educational outcomes. Outcomes may include graduation rates, college-going rates, income after high school, health and well being, but test scores are, for us here at OEP, an important one.

*side note here to school district folks- be sure to carefully check your graduation rate data from ADE- we hear it may be more ‘messy’ than in prior years*

Straight comparisons between PARCC test scores for Arkansas students and kids in other states may be inappropriate given differences in student demographics, but there are processes to adjust for poverty and compare school performance across states and even nations (check out schoolgrades.org). And – newsflash- there is never a PERFECT comparison for any kids or school or state, but we still need to try to responsibly compare outcomes and find out what is working to get students learning and ready for a successful life after school.

In today’s world of powerful data and analysis techniques, there are responsible ways to compare without needing to use asterisks. Here at OEP we know that kids and schools and states all face different challenges, and make every effort to focus the conversation not just on ‘highest test score’ but on growth and improvement as well. Just comparing within Arkansas is tricky, but these conversations are more complicated when states are using different tests. PARCC assessments were intended to facilitate constructive comparison of student performance across state lines, but varied testing patterns mean we still need some asterisks.  Hopefully, we won’t need any asterisks next year to compare student performance in Arkansas to Alabama (the only other state using ACT Aspire).

SO- Senator Clark and parents across the state, here at OEP we feel your frustration. Unfortunately, we can’t change the education political landscape and have all kids in the country take the same test.  As policymakers and parents, however, we can change the conversation and ask our schools- what information do you have about how our students are learning and growing compared to peers across the country?

Because in real life- there aren’t any asterisks.asterisk

 

 

PARCC Results for Schools and Districts

In The View from the OEP on January 20, 2016 at 2:10 pm

Although many parents have not yet received their student’s PARCC scores, last week school- and district- level PARCC results were released by the ADE.  It is important to note that results may be adjusted slightly once data appeals from districts are incorporated, but the preliminary information provides us here at OEP an opportunity to begin examining the scores!

The ADE released the information for each grade level, but we compiled the results to look at overall school and district performance. The data are available here on our website:

Algebra II and 11th grade literacy End-Of-Course results were not included to the school and district level summaries.  Why?  Because these assessments were optional last year, and so should not be used for comparative purposes. (Keep an eye out for this error when you consider sharing the school-ranking sites you see on Facebook!)

We have again calculated our ‘OEP GPA’ for each school.  Similar to grades that students receive, this indicator that gives the most credit (4 pts) to students who score at the highest level, and no credit to students who score at the lowest. We think this is a more informative value than overall percent meeting standards, because it rewards schools for student performing at the highest levels, and also provides points for students who are nearing expectations.

Some may wonder why we are talking about a test that has already been abandoned, but we want to be sure we don’t miss an  opportunity to learn something about how our schools are performing.  Although students will be taking a new test- the ACT Aspire- in just a few short months, here at the OEP we want to acknowledge the resources spent developing and taking the PARCC tests and learn whatever we can that may help improve education for our students.

Although it can be confusing to compare to prior performance on different tests,  here are some preliminary observations:

Proficiency rates are down.

From the state-level scores released in the fall, we knew that fewer students Met or Exceeded Expectations on the PARCC than had been identified as Proficient or Advanced on the prior exams exams. Now we can see that this pattern is true for schools and districts as well.

Does that mean our schools are doing worse than before?  Not necessarily. Since we are using different assessments, it can be difficult to determine if districts and schools are doing ‘better or worse’. Stay tuned for information about which schools demonstrated higher performance on the PARCC exams than we would have predicted!

Performance follows past trends.

Schools that were traditionally high performing on the previous exams scores relatively well on PARCC too.  This isn’t surprising – given that the assessments are still measuring academic skills in mathematics and English Language Arts. Although the expectations were higher, schools performing at the top are typically still at the top.

Does that mean schools didn’t improve?  No!  In fact, OEP will highlight specific schools that demonstrated large improvement from prior years.

Performance is associated with other factors too.

Just like on the prior assessments, schools with greater percentages of students participating in Free/Reduced Lunch programs (FRL) tend to have lower performance. The magnitude of the correlation (r= -0.63) is similar to prior years. This indicates that although performance declined, the PARCC tests were more difficult for all districts, not just districts with high FRL rates.

Does that mean students participating in FRL programs can’t meet standards?  No!  Of course not!  As we dig deeper into the data, OEP will highlight schools that are “Beating the Odds” and achieving high proficiency with their at-risk students.

It was the first (and last) time for PARCC tests, but we want to be sure we don’t miss an  opportunity to learn something about how our schools are performing. We will dig in and share what we learn- and if YOU have questions you want answered just let us know!

Stay tuned here for more information about the PARCC results!

Quality Counts 2016

In The View from the OEP on January 7, 2016 at 11:02 am

Today Education Week released their 20th annual Quality Counts report, which grades each state on their education performance.  This year Arkansas received an overall grade of C- and is ranked 41st overall.  Arkansas received a C- last year as well, but was ranked slightly higher nationally at 36th.  As we have discussed in previous blog posts there are several issues with the grading system, and we cannot compare overall scores prior to 2015, because the grading criteria was different in earlier reports. Today we wanted to dig into the grades and see what information might be useful.

This year’s report includes summative grades and rankings for states on education indicators as well as a special focus on school accountability, specifically assessing student achievement during the No Child Left Behind era.

What is Being Graded?

A state’s overall grade is the average of its scores on three separate indicators: Chance-for-Success, K-12 Achievement and School Finance.  Prior to 2015 there were six indicators, however the format was updated in 2015 in an attempt to focus on “outcomes rather than on policy and processes.” Although the report can be useful there are several issues with the grading system; previous blog posts have discussed the flawed nature of the grading system. 

While we cannot compare overall scores prior to 2015, because the grading criteria was different in earlier reports, we can compare performance in the categories retained for the 2015 and 2016 reports: Chance for Success, School Finance and K-12 Student Achievement.

Here are the most recent three years of  Arkansas grades in each of the categories considered for 2016.  The full report highlighting Arkansas student achievement can be accessed here.

2016 Quality Counts Categories AR Grade 2014 AR Grade 2015 AR Grade 2016
Chance for Success C- C-  C-
School Finance C C  C
K-12 Achievement D+ D+  D

The High and the Low:

Each of the three areas are an average of many other scores, so here at OEP we wanted to bring out the high (ranked in top 15 states) and low (ranked in bottom 15 states) areas for each category.

Chance for Success: According to EdWeek, “The Chance-for-Success framework allows states to identify strong and weak links in their residents’ educational life course―their typical trajectory from childhood through adulthood.”

  • Strong scores:  steady adult employment. Note that students attending preschool was also above the national average and ranked 18th!
  • Low scores: family income, parental education, middle school math, postsecondary enrollment, annual income and adult educational attainment

K-12 Achievement:  The K-12 Achievement Index examines 18 distinct state achievement measures related to reading and math performance, high school graduation rates, and the results of Advanced Placement exams.

  • Strong scores:  8th grade math gains and closing of the 4th grade reading achievement gap
  • Low scores: achievement levels, math excellence, family income, parental education, middle school math, postsecondary enrollment, annual income and adult educational attainment.

School Finance: Examining school finance can provide insight into how a state is supporting public education.

  • Strong scores: Equity of spending, and spending on education
  • Low scores: not spending as much as the national average

The strong scores summarized above reflect Arkansas’s commitment to education and that students are making gains.  The low scores reflect the many challenges that Arkansas students face: poverty, low parental and adult educational attainment, and, unfortunately, low achievement (especially in math). Although a D in student achievement is a decrease from last year’s D+ and definitely nothing to brag about, it is important to remember that Arkansas’ students are making gains.

If policymakers and education leaders can focus on meaningful data, like growth and efficiency in the face of disadvantage, rather than an overall grade, then students in Arkansas can continue to improve and reach greater levels of educational and lifelong success.

Take THAT Massachusetts

In The View from the OEP on December 16, 2015 at 11:33 am

0402_graduation_caps_416x416

Data released yesterday by the U.S Department of Education show that Arkansas’ high school graduation rate has increased AGAIN!  Not only is Arkansas graduating more students than the national average, but also graduating more students than Massachusetts (a state known for high performance)!

Seriously!  Take THAT Massachusetts!

How Do Arkansas Graduation Rates Compare to the Nation?

There’s a big graduation advantage for Arkansas students.  For the first time, more than 80% of African American students graduated.  That’s 9 percentage points higher than the national average. Arkansas also has a higher Hispanic rate- 85% compared to 76%, and even white students are more likely to graduate in Arkansas compared to their peers nationally.

Students who are Economically Disadvantaged are also more likely to graduate in Arkansas, and students who are Limited English Proficient or have Disabilities have a 20 point advantage over their peers nationally.

WOW!

ar

 

What About Graduation Gaps?

GAP.001

Differences in graduation rates for students from different backgrounds are closing, and because all graduation rates are increasing, which is a good thing!  The gap between Black and White student graduation rates has closed 33% since 2010, from 12 to 8 points.  Nationally, the gap has been slower to close, decreasing only 12% and reflecting a 15 point gap this year.

The Arkansas data reflects a dramatic closure in the gap between Hispanic and White students.  The gap has closed 60% since 2010, from 12 to 5 points.  Unfortunately, the national Hispanic/ White graduation rate gap remains essentially unchanged at 11 points this year.

In terms of economically disadvantaged students, the data only allow us to examine the gap between the graduation rate for these students and the overall graduation rate. We see good news here as well- with the gap closing to only 4 percentage points- a decline 0f 33%. Because the Economically Disadvantage students are included in the overall graduation rate, however, we look forward to Arkansas’ more appropriate comparison between Targeted Achievement Gap Group (TAGG) student graduation rates and their Non-TAGG peers.

How About the Neighbors?

 

Arkansas is graduating students at rates similar to states that have fewer students at risk, and has the largest gains in overall graduation rates of the bordering states.  Arkansas’ overall graduation rate has increased 6 points since 2011, and although 61% of students in Arkansas are eligible for Free/Reduced Lunch, the graduation rate of 87% matched that of Missouri where only 45% of students are economically disadvantaged.

Screen Shot 2015-12-16 at 9.44.10 AM

And, as we mentioned, Arkansas for the first time reports a higher graduation rate than Massachusetts (87% to 86%).  In fact, we rank 15th in the nation.  Hooray for Arkansas!

What does it MEAN for Arkansas students?

 

We know kids who graduate from high school perform better in life: there are significant financial consequences for students who do not graduate from high school. On average, graduates earn more than $10,000 more per year than their peers who didn’t receive a diploma.

While we celebrate the rising graduation rates, there are some who are concerned that graduation rates might be an imperfect metric of student success.  Other metrics show Arkansas students are not well prepared for success.

ACT scores have remained virtually unchanged even though graduation rates are rising, and only 21% of students in the class of 2014 met College-Ready Benchmarks on the ACT.

In addition, a report from the Arkansas Department of Higher Education highlights that 35.2% of students who had just graduated from high school were required to complete remedial coursework once they entered college. Although that is the lowest remediation rate since 2009, it is still over a third of high school graduates entering college unprepared.  The report also noted that 21.4% percent of the students required to take remedial courses had graduated high school with a 3.0 GPA or better.

Graduation Rates are Not Going Away

 

The Every Student Succeeds Act continues to place emphasis on graduation rates. The new federal education law, signed by President Barack Obama last week, requires states to intervene in high schools that failed to graduate more than two-thirds of their students, commonly referred to as “dropout factories.”  Arkansas has very few schools with graduation rates lower than 66%, and those that do serve unique student populations.  You can access the list of 2013-14 graduation rates for Arkansas schools and districts here.

Here at the OEP, we celebrate Arkansas’ success in improving high school graduation rates!  We also suggest that policymakers, parents and students keep working, and remember that a high school diploma is only one of the keys to student success.

 

 

House and Senate Education Committees Meet

In AR Legislature, The View from the OEP on December 16, 2015 at 11:32 am

capital-pic

The Arkansas House and Senate education committees met jointly this week and spent the most time talking about how to address differences in transportation costs across school districts.

Student Transportation

Education committee members discussed the ongoing struggle to fund transportation costs equitably across school districts. The current method of per-student funding for transportation does not account for such differences as actual number of riders, population density, bus route miles, geographic terrain, and road conditions. This creates what committee members called the “haves,” those districts where transportation costs are below their funding allocations, and the “have-nots,” where high transportation costs exceed funding.

The Bureau of Legislative Research (BLR) found in a 2014 study that a formula including ADM, actual riders, and daily route miles explained 98% of transportation costs, compared to ADM alone which explained 79% of costs. Despite the hard work of the education committees in the 2015 regular session to use this evidence-based formula in public school funding legislation, Act 1248 included it only in temporary language. The accompanying $3 million appropriation for supplemental funding is in Category C, which means there will not be enough time to distribute the money in 2015-16.

Committee members wrestled with whether and how to address the supplemental funding issue in the 2016 fiscal session. The committees also grappled with the possibility that dramatic changes in costs in the Little Rock and North Little Rock school districts may occur after 2016-17 and alter the transportation funding landscape altogether.

Niche Rankings Need Some Salt

In The View from the OEP on December 2, 2015 at 10:20 am

SALT

Last week, Facebook was full of ‘shares’ of the Niche district rankings.  While we love to see local communities celebrate their school systems, here at OEP we are concerned about the methods used and the information included in the rankings.  We feel the Niche rankings need to be taken with more than a grain of salt.

Niche claims, “By providing reviews and insight from everyday experts, we make choosing a neighborhood, college, or K-12 school a more transparent process.”  They clearly have a different idea than we do about what transparency means.

 

What is the Niche ranking based on?

 

Clicking on read the full methodology (yes, we realize we are the ONLY ones who would do this) provides a general description about the process and an assurance that Niche ‘goes to great lengths’ to ensure their rankings represent a comprehensive assessment of each district.

Further digging lead to some more information about how the ‘best districts’ were determined.  This includes:

 

Sounds reasonable- What’s the problem?

 

We like the idea of viewing schools as more than just their academic scores BUT it is important that each aspect of the grade be valid and reliable.  Do you see a trend in the descriptions of each aspect?  We do- “student, alumni and parent survey responses.”

We think parent and student feedback to schools is great and can be very informative.  In fact, some schools in California are using such data as a part of their school accountability process.  Building survey instruments that really measure what you are interested in is tricky, however, and it can be even trickier to use the results appropriately.  We can’t find the Niche survey, so can’t weigh in on it’s quality, but we don’t like the way they used  – “student, alumni and parent survey responses” to grade a school.  We worry about this for three reasons:

  1.  There’s no way to know if the folks answering the survey even attend that district – this is just self-reported
  2. The sample is self-selected (meaning there could be a bias in what types of people are responding) and
  3. The sample can be very small. One of the Top 10 districts in Arkansas had only 19 reviews since 2013.  These “19” were comprised of smaller parts of reviews by  just 6 people: 5 high school seniors and 1 “Niche user”.

The survey information is weighted heavily in most areas (you can see the details of each ‘grade’ by clicking on the links for each aspect above) and accounts for 30% of a school’s overall grade.  Every opinion has value BUT… this is unreliable data.

Speaking of unreliable data- a lot of the data used by Niche is incorrect, based on self-report, or used multiple times in different rankings.  According to Niche, Fayetteville School District had 0% of students pass an Advanced Placement test.  According to the ADE, 67% of Fayetteville students passed AP tests.  Which do you think is correct?  The ‘Top College Score’ – which contributes 15% of the Best Academics grade- is based on the Niche rankings of the colleges that Niche users report ‘being interested in’- nothing to do with where students actually go to college.  The average ACT/SAT score is similarly self-reported and, not surprisingly, different than the values reported by the state department. I could go on…

We have LOTS of issues with the Niche rankings, not the least of which is all the pop-up ads, but the main one is a lack of true transparency about what is included in the ranking and how it is calculated.  Although the categories are listed, there is no way to download the data for verification and analyze the models (again- we realize we are the ONLY people who would do this- don’t judge- it’s what we do).

 

Is there anything useful in the ranking?

 

From a parent perspective, it may be interesting to see what people say about the district, but I would take it with a grain (or two) of salt.   Similarly, from a school district perspective, it may be informative to check and see what your students, alumni and parents are saying.  If I were a superintendent, I might try to get LOTS of people to go on and say good things about my schools!  A better move, however, would be to conduct a local parent, student and teacher survey, asking the questions I really want to know about and sharing the results. If you are interested – OEP can help you develop a good one!

We like how Niche pulls a lot of information about a school district together in one place, but don’t like the quality of the information or the fact that their calculations can’t be verified.  This highlights another important point- that it is very complicated to compare school systems across states, which Niche does. The data are inconsistent (tests vary from state to state) and come from a lot of different sources.

 

What’s the harm?

 

It may seems harmless to share your district’s Niche ranking on Facebook- but consider that it adds noise to the ongoing and difficult discussion regarding school quality.

YES, we want stakeholders to have information about how their local school is performing, and YES, it is important to consider aspects other than just academics, but BAD DATA CAN BE WORSE THAN NO DATA.

We like the idea behind a simplified ranking, but Niche misses the mark. It’s not like looking at rankings to choose a hotel for a weekend stay,  schools are supported by our own dollars and are a major influence on the lives of our children.

Arkansas is working hard to refine the process for evaluating schools.  In the meantime, if you want to know if your school district is great, you can find high quality data and analysis on our website and we recommend you go visit your local school district! While you are there- ask about growth data, how they know if they are making a difference for kids, and if they have conducted a meaningful survey of parents, students and teachers.

 

 

 

 

 

What’s the Deal with NAEP Results?

In The View from the OEP on November 18, 2015 at 11:49 am

kidmath2

Today we dig more deeply into the recently released National Assessment of Educational Progress results- examining how Arkansas students scored compared to neighbor states and other states that adopted, and did not adopt, Common Core State Standards.  For more detailed information read the full policy brief.

Two weeks ago we discussed that Arkansas’ scores had declined somewhat across the board,  and that 4th grade Math scores had declined significantly since the last NAEP assessment in 2013.

That got us to wondering- why? So we asked a series of questions:


Was it just us? No, but in our neighborhood- yes.

 

A significant decline in scores feels bad, but we feel better is we aren’t declining alone.  Nationally, there was a decline in 4th grade Math scores as well, but not for our neighbor states.  We examined the 4th grade Math performance of Louisiana, Mississippi, Missouri, Oklahoma,  Tennessee and Texas, and found that while our scores decreased, on average their scores increased!  For the first time since 2000, Arkansas’ border states surpassed our performance in 4th grade Math.


Was it because of Common Core? It’s hard to tell, but probably not. 

 

Common Core State Standards (CCSS) have been a much debated topic over the past year, and are currently being revised to ensure they meet Arkansas’ needs.

Pursuing the question of whether or not the full implementation of CCSS in 2014-15 could have impacted Arkansas’ performance on the NAEP, we compared Arkansas with other states that have implemented the standards as well as those that have not (a full list is in the policy brief).

As seen in Figure 1, non-CCSS states did not experience a decline in 4th grade Math scores in 2015.  Other states that implemented CCSS experienced a slight decline, but it was not as significant as Arkansas’.

Figure 1: NAEP Mean Scaled Score for 4th Grade Math: Arkansas, CCSS States and Non-CCSS States, 2003-2015CCSSvNon2

The fidelity and quality of CCSS implementation varied across states, and perhaps elementary math standards were particularly difficult for Arkansas teachers to implement effectively. We must conclude, however, that Common Core State Standards are not the sole contributor to this decline in Arkansas’ performance.


Were these 4th grade students lower math performers on other tests too? Not really. 

 

Here at the OEP we always use multiple pieces of data.  Perhaps the students who took the 4th Grade NAEP in 2015 were just academically lower achieving than students who had completed the NAEP in previous years.  To examine this question, we considered the 3rd grade Math Benchmark performance for the years prior to the 4th grade NAEP administrations.  Although not exactly the same students, because only a representative sample complete the NAEP, we can assume that the students are essentially the same in regard to characteristics that would impact academic performance.

Figure 2: Arkansas 3rd Grade Benchmark Math Percent Proficient and Subsequent 4th Grade NAEP Math Percent Proficient,2005-06 through 2013-14

NAEP BM

The percentage of 3rd grade students scoring Proficient/ Advanced on the Math Benchmark steadily increased between 2005-06 through 2011-12, but shows a 3% decline between 2011-12 and 2013-14. The subsequent 4th grade Math NAEP exams reflected slight increases in proficiency over time, but  a steeper decline of eight percentage points from 2013 to 2015. The trend in the Benchmark patterns of the years are similar to that of the NAEP exams but does not present us with a cause for Arkansas’ significant decline in 4th grade Math performance on the 2015 NAEP.  The scores of 3rd grade students on the Arkansas Benchmark exam in 2013-14 did not indicate there would be such a significant decline on the 4th grade NAEP.


So What Should We Do?  Get More and Better Information! 

 

Digging into assessment results often leads us to more questions.  Unfortunately, it isn’t clear why the NAEP 4th grade Math scores declined so significantly between 2013 and 2015.  Recently released PARCC results, however,  indicate that fewer than one in four 4th graders scored proficient in math– that’s even lower than the NAEP results (see Table 1).

Table 1: PARCC and NAEP Percent Proficient, by Grade and Subject, 2014-15

PARCCNAEP

This spring, Arkansas students will take a new assessment, the ACT Aspire, making comparisons between states and from one year to the next difficult.  NAEP for 4th and 8th grade will not be administered again until 2017, which is a long time to wait to see if this year’s results were a minor blip or the beginning of a larger decline.  Our students can’t wait.

Many districts throughout the state are not waiting.  They are using high quality interim assessments that compare students to peers nationally.  Not waiting until 3rd grade to see how students are performing, many use short interactive assessments as early as kindergarten to ensure that students have a strong foundation. Not waiting for state assessment data to identify students in need of support, because these assessments can help schools understand where students are- not just those students that are behind, but also students that are ahead of their peers and can benefit from enrichment.  If we all don’t wait another day, and use data to make the most of every learning moment, and make sure that students are getting the instruction they need to grow, hopefully Arkansas NAEP scores will rebound in 2017, and our students will be more prepared for success.

 

 

 

Say Goodbye to the 70s- PARCC Scores are here

In The View from the OEP on November 12, 2015 at 1:57 pm

travoltaSeems like just yesterday it was the 70s:

in 2013-14, 78 percent of students scored proficient or advanced on state literacy assessments, and 72% scored proficient or advanced on state math assessments.

Today the State Board of Education approved the PARCC cut off scores for grades 3-8 English Language Arts and Math, allowing us the first opportunity to see how well Arkansas students scored on the new, much discussed, and now abandoned test.

How Did Arkansas Do?

Table 1. Percent of Arkansas Students Scoring “Proficient” (Level 4) and Above on 

2014-15 PARCC Assessment

PARCC

WHOA! Only about one in three Arkansas students scored proficient or better?

Last year’s test data showed more than twice that amount!  WHAT HAPPENED?

We got a new test!

PARCC is the first assessment aligned to Arkansas’ Common Core State Standards, which set a higher bar for student learning, emphasizing the need for students to demonstrate critical thinking, problem solving, and clear writing.  PARCC results cannot be compared with the earlier Arkansas Benchmark results, both because this is a new test and a different test. This will be the only year of PARCC results, as Arkansas switched to ACT Aspire for assessment this school year.

In fact, these proficiency rates might sound familiar to you.  That is because just two weeks ago the results of the on the National Assessment of Educational Progress (NAEP), were released, and the scores were very similar.

How Did Arkansas Compare to Other PARCC States?

You may remember that  one of the key benefits of PARCC was that we would be able to compare Arkansas student performance to the performance of students in other states.

So far seven states have released their scores for grades 3-8.  Note: Some additional states have released high school scores, but because of differences in testing requirements and implementation, cross-state comparison of high school results isn’t useful.

The six other states that have released (at least preliminary) PARCC results are New Mexico, Louisiana, Illinois, Ohio, New Jersey and  Massachusetts.  The states are VERY different in many ways, but a key characteristic related to assessment is poverty. We would expect states with enrolling a greater percentage of students who are eligible for Free/Reduced Price Lunch (a proxy variable for poverty) will underperform states with fewer students eligible for Free/Reduced Price Lunch.  The seven PARCC states that have released scores range in FRL percentages, from New Mexico, with the greatest poverty at 68.5% of FRL students, to Massachusetts, with only 35.1% of FRL students. In the figures below, states are arranged from MOST FRL on the left to LEAST FRL on the right.  Not surprisingly, Massachusetts outperformed New Mexico.   Arkansas enrolls 60.9% of students eligible for FRL and is represented in the figures below by the RED bars.

Figure 1. Percent of Students Scoring “Proficient” (Level 4) and Above on 

2014-15 PARCC ELA Assessment

PARC ELANote- Ohio did not report scores for 3rd grade ELA

Figure 2. Percent of Students Scoring “Proficient” (Level 4) and Above on 

2014-15 PARCC Assessment

PARCC Math

Notes- New Jersey 8th grade scores are not representative.  Massachusetts allowed districts to choose between PARCC and the prior state assessment, and the split was fairly even.  Reported PARCC results for MA are based on a large representative sample, matched on achievement and demographic variables prior to score availability.

What Does This Mean?

English Language Arts

  • Arkansas is performing similarly to what we might expect, given our student population. Arkansas students outperform students from New Mexico, and are not as likely to be proficient as students from Massachusetts.
  • In many grades, Arkansas students scored similarly to students from states which have less disadvantaged student populations (Illinois and Ohio).
  • Interestingly, Louisiana students outperformed Arkansas students in almost every grade, even though they are more likely to be disadvantaged.

Math

  • Arkansas is performing similarly to what we might expect, given our student population. Arkansas students outperform students from New Mexico, and are not as likely to be proficient as students from Massachusetts.
  • Interestingly, Louisiana students outperformed (or equaled) Arkansas students in every grade, even though they are more likely to be disadvantaged.
  • 8th grade math scores are variable, perhaps in part because some advanced students completed high school level assessments (Algebra or Math I) instead of 8th grade math.

How Does PARCC Compare to NAEP?

The scores are very similar, but there are some trends in relationships between the scores.

In Reading, PARCC proficiency rates are typically a little bit higher than NAEP. Arkansas 4th graders were 2 percentage points more likely to be proficient on PARCC, and Arkansas 8th graders were 5 percentage points more likely.

In Math, PARCC proficiency rates  are typically a little bit lower than NAEP at 4th grade, and quite varied at 8th grade. Arkansas 4th and 8th  graders were 8 percentage points less likely to be proficient on PARCC than on NAEP.

So What Now?

Arkansas has gotten a lot of feedback about it’s education system recently- and while it isn’t great news, we need to be sure we have the right takeaways as we continue to move Arkansas education forward.

  1. Face the Music: PARCC and NAEP scores give us a clear picture of how Arkansas students perform compared to other states.  Both assessments are sending the same message- about one in three Arkansas students are ‘on grade level’.
  2. Learn the Steps: Arkansas teachers, students and parents need frequent, high quality data to provide a clear picture of where students are academically.
  3. Practice the Moves: Teachers need training on how to EFFECTIVELY use assessment data for their students to transform the instruction in the classroom.
  4. Find a Partner: Arkansas should consider why Louisiana is consistently outperforming us.
  5. Strut Your Stuff!  We look forward to seeing Arkansas students demonstrate improved performance!  For comparable data, we will likely need to wait until NAEP 2017.
Follow

Get every new post delivered to your Inbox.

Join 256 other followers