University of Arkansas Office for Education Policy

Niche Rankings Need Some Salt

In The View from the OEP on December 2, 2015 at 10:20 am

SALT

Last week, Facebook was full of ‘shares’ of the Niche district rankings.  While we love to see local communities celebrate their school systems, here at OEP we are concerned about the methods used and the information included in the rankings.  We feel the Niche rankings need to be taken with more than a grain of salt.

Niche claims, “By providing reviews and insight from everyday experts, we make choosing a neighborhood, college, or K-12 school a more transparent process.”  They clearly have a different idea than we do about what transparency means.

 

What is the Niche ranking based on?

 

Clicking on read the full methodology (yes, we realize we are the ONLY ones who would do this) provides a general description about the process and an assurance that Niche ‘goes to great lengths’ to ensure their rankings represent a comprehensive assessment of each district.

Further digging lead to some more information about how the ‘best districts’ were determined.  This includes:

 

Sounds reasonable- What’s the problem?

 

We like the idea of viewing schools as more than just their academic scores BUT it is important that each aspect of the grade be valid and reliable.  Do you see a trend in the descriptions of each aspect?  We do- “student, alumni and parent survey responses.”

We think parent and student feedback to schools is great and can be very informative.  In fact, some schools in California are using such data as a part of their school accountability process.  Building survey instruments that really measure what you are interested in is tricky, however, and it can be even trickier to use the results appropriately.  We can’t find the Niche survey, so can’t weigh in on it’s quality, but we don’t like the way they used  – “student, alumni and parent survey responses” to grade a school.  We worry about this for three reasons:

  1.  There’s no way to know if the folks answering the survey even attend that district – this is just self-reported
  2. The sample is self-selected (meaning there could be a bias in what types of people are responding) and
  3. The sample can be very small. One of the Top 10 districts in Arkansas had only 19 reviews since 2013.  These “19” were comprised of smaller parts of reviews by  just 6 people: 5 high school seniors and 1 “Niche user”.

The survey information is weighted heavily in most areas (you can see the details of each ‘grade’ by clicking on the links for each aspect above) and accounts for 30% of a school’s overall grade.  Every opinion has value BUT… this is unreliable data.

Speaking of unreliable data- a lot of the data used by Niche is incorrect, based on self-report, or used multiple times in different rankings.  According to Niche, Fayetteville School District had 0% of students pass an Advanced Placement test.  According to the ADE, 67% of Fayetteville students passed AP tests.  Which do you think is correct?  The ‘Top College Score’ – which contributes 15% of the Best Academics grade- is based on the Niche rankings of the colleges that Niche users report ‘being interested in’- nothing to do with where students actually go to college.  The average ACT/SAT score is similarly self-reported and, not surprisingly, different than the values reported by the state department. I could go on…

We have LOTS of issues with the Niche rankings, not the least of which is all the pop-up ads, but the main one is a lack of true transparency about what is included in the ranking and how it is calculated.  Although the categories are listed, there is no way to download the data for verification and analyze the models (again- we realize we are the ONLY people who would do this- don’t judge- it’s what we do).

 

Is there anything useful in the ranking?

 

From a parent perspective, it may be interesting to see what people say about the district, but I would take it with a grain (or two) of salt.   Similarly, from a school district perspective, it may be informative to check and see what your students, alumni and parents are saying.  If I were a superintendent, I might try to get LOTS of people to go on and say good things about my schools!  A better move, however, would be to conduct a local parent, student and teacher survey, asking the questions I really want to know about and sharing the results. If you are interested – OEP can help you develop a good one!

We like how Niche pulls a lot of information about a school district together in one place, but don’t like the quality of the information or the fact that their calculations can’t be verified.  This highlights another important point- that it is very complicated to compare school systems across states, which Niche does. The data are inconsistent (tests vary from state to state) and come from a lot of different sources.

 

What’s the harm?

 

It may seems harmless to share your district’s Niche ranking on Facebook- but consider that it adds noise to the ongoing and difficult discussion regarding school quality.

YES, we want stakeholders to have information about how their local school is performing, and YES, it is important to consider aspects other than just academics, but BAD DATA CAN BE WORSE THAN NO DATA.

We like the idea behind a simplified ranking, but Niche misses the mark. It’s not like looking at rankings to choose a hotel for a weekend stay,  schools are supported by our own dollars and are a major influence on the lives of our children.

Arkansas is working hard to refine the process for evaluating schools.  In the meantime, if you want to know if your school district is great, you can find high quality data and analysis on our website and we recommend you go visit your local school district! While you are there- ask about growth data, how they know if they are making a difference for kids, and if they have conducted a meaningful survey of parents, students and teachers.

 

 

 

 

 

Advertisements
  1. Thank you! I wondered where the information came from with regard to most of the posted grades that led to the rankings. To rely on self-reporting and surveys for grading schools is weak, at best. So perceptual data tells me that one school is much better than another which may or may not mean the school is, in fact, any better, just that people who felt good about the school were the ones who primarily answered the surveys (or harsh criticism was excluded).

    • Thank you for your comment and we are glad the information was helpful. Yes, when people self-select to provide reviews on a school district there is no way of knowing if the thoughts and opinions represent what all people typically feel about the district. In order to get reliable information, you need a random selection and enough responses that the data will be stable. This is more difficult than having folks submit reviews online, but would be more valuable information.

Comments are closed.

%d bloggers like this: