University of Arkansas Office for Education Policy

Where The Metrics are Made Up and the Points Don’t Matter: The School Quality and Student Success Indicator in ESSA

In The View from the OEP on August 30, 2017 at 1:08 pm

Earlier this month, the third draft of the Arkansas’s ESSA plan was submitted to Gov. Asa Hutchinson for review. Given the focus on a successful beginning of school, many educators likely haven’t had time to examine this new draft, but feedback is due by August 31st, and we think you might want to give some on the plethora of measures now included in the School Quality and Student Success Indicator. Especially if you are a high school administrator (please share this blog).  We will submit our suggestions (at the end of this post) today to ade.essacomments@arkansas.gov and we suggest you send yours as well!

Reading through the updated info about the School Quality and Student Success Indicator (SQSS) we were reminded of one of our favorite shows: Whose Line Is It Anyway?

 

Points don't matter

 

Our last post about the ESSA plan was 3 months ago after the 2nd draft was released.  We had served on the Accountability Advisory Team and were pretty pleased with the resulting plan.  In that blog, we encouraged stakeholders to submit feedback about the plan.  Unfortunately, it seems like there were too many ‘suggestions’ about the School Quality and Student Success Indicators, because now instead of 2 measures of school quality and student success there are 11.   Keep in mind that this is in addition to the other indicators included in the plan: academic achievement, academic growth, graduation rate, and progress in achieving English Language Proficiency.

Here at the OEP, our view is that the inclusion of so many measures will make it more difficult for school leaders to determine if their school is meeting expectations for school quality and student success. We think ADE needs to reduce the number of measures being included in the School Quality and Student Success Indicator.  A few clear indicators will be better for our schools and students.

It’s kind of like if your doctor told you to eat healthier.  Eating healthier could mean different things to different people, depending on if you are needing to lower your cholesterol, manage your diabetes, lose weight, or even put on some pounds.  Tracking your calories, fats, sodium, potassium, protein, carbohydrates, sugar, iron, folic acid, and vitamins A and C, will give you a lot of information that may or may not make you healthier, and will likely make you give up altogether on your attempts to change your diet for the better. What might work, though, is if your doctor gave you a clear goal that is easy to keep track of, like “Eat more veggies” or “Stay under 40 grams of carbohydrates each day”.


What SQSS Was Before: As we discussed in our earlier blog about Arkansas’ ESSA plan, the first two versions of the ESSA plan included some of these clear goals in the form of measures for the School Quality and Student Success Indicator. As you can see in Table 16 from the 2nd draft, there were one or two indicators per grade span.

5th Indicators

For ‘elementary schools‘, those enrolling a majority of their students in Kindergarten through 5th grades, the two measures were the percent of students who attended school at least 90% of the time (NOT Chronically Absent), and the percent of students ‘Reading Ready’ in 3rd grade (measured until the new K-2 assessments get implemented as the percent of 3rd grade students meeting or exceeding expectations on ACT Aspire reading).

For ‘middle schools‘, those enrolling a majority of their students in 6th through 8th grades, the two measures were the percent of students who attended school at least 90% of the time (NOT Chronically Absent), and the percent of students meeting or exceeding expectations on ACT Aspire science.

For ‘high schools‘, there was only one measure: the percent of graduated with one or more AP/IB/or concurrent credit.

Pretty straightforward: get kids to come to school, be ready to read by grade 3, make sure they are learning science in the middle grades, and make an effort to have students take rigorous courses in high school. We appreciated how these measures were easily collected and easily interpreted.

We liked the third grade on-level reading because of the focus it will put on early reading progress in K-2, and we also liked the fact that science is being included somewhere in the metrics.

We had concerns with chronic absenteeism because the metric may reflect events at home rather than the quality of the school. If I leave for work before the bus comes so I can’t help my 1st grade student get to the bus on time, or if I need an older sibling to stay home with the younger siblings if they are sick, my student is absent due to issues OUTSIDE of school rather than issues of school quality. Could schools implement programs and supports to help my family get to school more consistently?  Sure, but it may be difficult to impact the wide variety of issues that some students and their families may face, and we aren’t convinced that absenteeism is a valid and reliable measure of school quality.

We also have concerns with only examining the percentage of graduates with ‘advanced’ coursework. First of all, just taking a course is not a complete indicator of success.  Arkansas invests to make AP tests FREE for all Arkansas students, so we suggest that the percentage of graduates that ‘pass’ the AP test may be a better indicator of school quality and student success.  In addition, we would have liked to see inclusion of other indicators of post-high school readiness: CTE program completers or the percentage of students receiving industry certifications.

 

Vision

 

Both chronic absenteeism and the ‘advanced coursework’ measures are particularly odd in the face of the Vision of the ADE and the tone of the ESSA plan.  Both the Vision and the plan talk about student-focused learning, but these measures are not student-focused at all. Although we didn’t love all of the metrics initially included int the School Quality and Student Success Indicator, they are not BAD measures, they are easy to collect and understand, there and only a max of 2 at each school which makes it manageable, and the points are 10% or less overall score (MOST OF WHICH IS BASED ON STUDENT GROWTH- HOORAY!).

 


What SQSS Is Now: In the the latest ESSA draft, however, the School Quality and Student Success Indicator,  is a mess.  The plan notes “The School Quality and Success Indicator was a focus of significant stakeholder feedback during the public comment period.”As you can see in Table 15 from the 3rd draft, there are now a lot more indicators of school quality and student success.

SSI indicators_version 3

 

Holy Moly!  That’s a lot of points and way more confusing than before! And now it is a greater percentage of the overall school score (15%).

Note that the indicators are now reported for grade levels instead of by the grade span if the school.  This is because all these indicators have to be combined at the student level.  You can read more about the super not-easy to calculate six-step process on page 43 of the ESSA plan.

For ‘elementary schools‘, the indicators still include chronic absenteeism, but the percent of students ‘Reading Ready’ in 3rd grade has been expanded to 3rd grade and up. We liked the 3rd grade indicator because we felt it would help focus on K-2 early literacy skills, but now we feel like this is just double-counting scores already used in the Achievement indicator.  In addition, science achievement and growth are included for elementary schools, which we don’t mind but it may adds more noise and district from the initial focus on early Literacy!

For ‘middle schools‘, the indicators still include chronic absenteeism, and the percent of students meeting expectations in science, but now also include science growth and the percentage of students meeting grade level expectations in reading.  We like including growth with science since achievement was initially included, but like with elementary- feels like double counting scores that are included in achievement.

High schools get hit hard with all these new metrics. Instead of just AP/ IB/ concurrent course taking there are now eleven measures

1) Chronic absenteeism is now a measure at the high school level, and while we still have concerns about the value reflecting things other than school quality, we do feel that in high school it may be more if a student’s decision to attend, and that most high schools could do more to get kids to come.

2 & 3) High schools are now, like elementary and middle schools, also examining science achievement and growth as a measure of school quality and student success for students through 10th grade.

4) High schools will also be measured on the percentage of students reading at grade level through 10th grade.

5 & 6) There are two measures for ACT for graduates: the high school receives a point for each student who scored at least 19 on the ACT (composite), and can get another half a point per student if they meet the college readiness subject benchmarks of 22 or over. We like including this measure because, as we have discussed, it has real meaning for students.  All 11th graders take ACT, and since students’ best score is used, maybe schools will work harder to promote all students taking it more than once. Criteria for scoring points for WorkKeys, the ACT career readiness assessment is being examined.

7) The GPA of graduates is another proposed measure of school quality and student success. This is the one indicator we really DO NOT LIKE. It seems to us that this may lead to pressure on teachers to give ‘better’ grades.  We know that grades are subjective and inconsistent from class to class and school to school. Grades are not a valid or reliable measure of school quality or student success.

8, 10 & 11) In addition to the AP/IB/ Concurrent indicator that was in the earlier drafts, now schools get points for students completing Service Learning courses and computer science courses.  Although we can see why these would be ‘on the list’ of measures for a quality school, we would still like to see a more career-focused indicator.

9) On-time credits is a new measure. We like the indicator that checks along the way for on-time credits, but aren’t sure how this looks in practice because it is new for Arkansas!

 

See what we mean about keeping track of too many indicators?

Herding-Cats

 


What SQSS Should Be:

We suggest ADE needs to reduce the number of measures being included in the School Quality and Student Success Indicator and the weight of the SQSS indicator in the Overall School Index should be pulled back to 10%.  A few clear, measurable indicators will be better for our schools and students than too many, but none of them are great measures of school quality or student success. Here’s what we would do:

“Reading Ready” just for 3rd grade.  We like the 3rd grade indicator because we feel it would help focus on K-2 early literacy skills, but when it is applied to all grades we feel like this is just double-counting scores already used in the Achievement indicator.

Science Achievement and Growth just for middle schools.  While we like the inclusion of science, we really need to have fewer, clearer measures. Elementary focus should be Literacy (reflected in part by “Reading Ready”) and Science is already included in the high school measures through ACT.

Combine all the course-taking measures into one, and give credit for any graduate that completed any of the listed classes. This is supposed to be about student-focused learning.  We want high schools to support students in pursuing their goals, not just enrolling in specific types of classes.

Leave out GPA. This is the one indicator we really do not like. It seems to us that this may lead to pressure on teachers to give ‘better’ grades.  We know that grades are subjective and inconsistent from teacher to teacher, class to class, and school to school. Grades are not a valid or reliable measure of school quality or student success and GPA should not be included in this indicator.

Start with on-time credits for just 9th grade, then add additional grades in subsequent years. Suddenly adding in this new measure at all grade levels (in addition to all the other measures) might be too much for the school to address at one time. A more measured approach would be less overwhelming for schools and could lead to better outcomes for students.

 

In summary- we think there should be fewer measures included in the School Quality and Student Success Indicator, and suggest that they should allocated as presented in the table below.

OEP’s proposed School Quality and Student Success Indicator Measures

OEP SQSS

We still want to see more career- ready indicators, and look forward to indicators that would be more representative of school quality, but this would be a WAY better start than what is in Draft 3. ADE makes it clear that these are not final measures: “This system will transition and improve over time as additional school quality and student success indicators are developed, validated, and used to replace or augment initially proposed indicators.”

Make your voice heard. Email your comments to ade.essacomments@arkansas.gov today or tomorrow (before August 31st)!

ADE will submit the final plan to the U.S. Department of Education in September 2017.

Federal feedback is expected by the end of the year, and ADE will implement aspects of the plan beginning in the 2018 − 19 school year.

Advertisements
%d bloggers like this: