Ken Pomeroy interview talks efficiency metrics, the NET rankings, and preseason rankings

[Vader_Storm]

Pretty decent interview here worth listening to if you are interested in understanding efficiency metrics. He talks about a lot of different stuff in this interview including how the NET rankings basically copy his rankings, and how scoring margin and running up the score is the most important thing in determining where you are ranked on his website and on the NET rankings which basically copy him. He also goes into detail about how his preseason rankings are established, how transfers, freshmen, and coaching changes impact his rankings, and how the preseason rankings also look at prior year history up to 5 years ago to rank teams. Interesting enough, he even talks about Pitt last season in this video.

In the end, this should put to bed the notion of just how important scoring margin is and running up the score against everybody is. Pitt and the acc need to run the score up in the non conference schedule game after game. Anyway, this is worth a listen to as I found some of the stuff in this interview quite interesting.

 

[MTkeeper23]

How can you talk about the flaws of the stat and stand by it? The difference between winning by 35 vs 30 is the same as the difference by winning by 6 or 1. How do they not taper off significantly when games are 20+ points apart. It’s a bunch of walk-ons and bench players chucking 3s at that point.

These are supposed to be statisticians.

 

[MarshallGoldberg]

How can you talk about the flaws of the stat and stand by it? The difference between winning by 35 vs 30 is the same as the difference by winning by 6 or 1. How do they not taper off significantly when games are 20+ points apart. It’s a bunch of walk-ons and bench players chucking 3s at that point.

These are supposed to be statisticians.

He said his ratings don't measure accomplishments and shouldn't be used for the tournament.

 

[Pitt3Putt]

He said his ratings don't measure accomplishments and shouldn't be used for the tournament.

It’s essentially a very rudimentary gambling model. Meant to be predictive.

It does do some adjustments for scoring margin and other factors.

NET I don’t think does any of that. That said, over the course of probably 2,000+ possessions in a season don’t think 10 points in garbage time here and there matters that much. And if you are getting to garbage time possessions on a frequent enough basis to be a large portion of your total season possessions, you’re almost certainly blowing teams out or getting blown out enough to establish if you’re good or bad.

 

[Vader_Storm]

It’s essentially a very rudimentary gambling model. Meant to be predictive.

It does do some adjustments for scoring margin and other factors.

NET I don’t think does any of that. That said, over the course of probably 2,000+ possessions in a season don’t think 10 points in garbage time here and there matters that much. And if you are getting to garbage time possessions on a frequent enough basis to be a large portion of your total season possessions, you’re almost certainly blowing teams out or getting blown out enough to establish if you’re good or bad.

1) The NET absolutely uses scoring margin just like Pomeroy and Torvik (which is my personal go to). In fact, you can actually watch teams rise and fall on these websites based game to game primarily on scoring margin. They go almost in sync with one another on game to game results even though each website has slightly different weights to rank teams. Sometime next year midseason for example. Pick a random game and a result of a blowout win against the spread. And watch the team rise significantly in all these efficiency websites together. They will make a similar jump on each efficiency site, including the NET. I actually proved this theory last year and Pomeroy just confirmed it in the interview above.

2) The NET doesn't seed teams based on the NET overall rankings. But it absolutely seeds teams with regards to Quad wins in the Q1 and Q2 pods, which is directly and indirectly related to the NET. It only takes a few bad teams in a conference to potentially crash the entire conference down the conference ranking system and limit high end Quad 1 and 2 wins. We saw that last year with Quinten Post and Boston College and how it wrecked the ACC as an example. We also saw it with Pitt last year as our non conference numbers were beyond awful and as we beat acc teams in conference we severely dragged the acc's numbers down and the conference down with it as we rose up.

3) You still need to run the score up, the whole league does. I agree with the poster above that the last 2 minutes in a blowout win should be irrelevant. I would even argue the efficiency ranking should be capped with a 20 point win when the walkons are on the floor at games end. But that's not how efficiency sites or the NET work. Games likes Alabama State last year, Michigan last year, and West Virginia last year killed our efficiency ranking and NET ranking and put us in a massive hole. It was almost a miracle we made the NCAA Tournament after that Alabama State game when you looked at the hole we put ourselves in.

 

[Pitt3Putt]

1) The NET absolutely uses scoring margin just like Pomeroy and Torvik (which is my personal go to). In fact, you can actually watch teams rise and fall on these websites based game to game primarily on scoring margin. They go almost in sync with one another on game to game results even though each website has slightly different weights to rank teams. Sometime next year midseason for example. Pick a random game and a result of a blowout win against the spread. And watch the team rise significantly in all these efficiency websites together. They will make a similar jump on each efficiency site, including the NET. I actually proved this theory last year and Pomeroy just confirmed it in the interview above.

2) The NET doesn't seed teams based on the NET overall rankings. But it absolutely seeds teams with regards to Quad wins in the Q1 and Q2 pods, which is directly and indirectly related to the NET. It only takes a few bad teams in a conference to potentially crash the entire conference down the conference ranking system and limit high end Quad 1 and 2 wins. We saw that last year with Quinten Post and Boston College and how it wrecked the ACC as an example. We also saw it with Pitt last year as our non conference numbers were beyond awful and as we beat acc teams in conference we severely dragged the acc's numbers down and the conference down with it as we rose up.

3) You still need to run the score up, the whole league does. I agree with the poster above that the last 2 minutes in a blowout win should be irrelevant. I would even argue the efficiency ranking should be capped with a 20 point win when the walkons are on the floor at games end. But that's not how efficiency sites or the NET work. Games likes Alabama State last year, Michigan last year, and West Virginia last year killed our efficiency ranking and NET ranking and put us in a massive hole. It was almost a miracle we made the NCAA Tournament after that Alabama State game when you looked at the hole we put ourselves in.

I meant that NET doesn’t make efficiency adjustments like KenPom.

It also wasn’t just the blowout losses that tanked the NET, it was all of the close ACC wins with few comfortable wins against non-UL opponents.

 

[Joe the Panther Fan]

1) The NET absolutely uses scoring margin just like Pomeroy and Torvik (which is my personal go to). In fact, you can actually watch teams rise and fall on these websites based game to game primarily on scoring margin. They go almost in sync with one another on game to game results even though each website has slightly different weights to rank teams. Sometime next year midseason for example. Pick a random game and a result of a blowout win against the spread. And watch the team rise significantly in all these efficiency websites together. They will make a similar jump on each efficiency site, including the NET. I actually proved this theory last year and Pomeroy just confirmed it in the interview above.

So what you are saying is that you made the astonishing discovery that teams that are really efficient in a game, which would result in them performing much better than expected, caused their rating based on efficiency to improve.

Who would have thought that would happen?

🤷‍♀️


I mean some people might have realized that that is exactly the way that ratings systems like that are designed to work right from the start, but I guess not everyone.

 

[Pitt3Putt]

So what you are saying is that you made the astonishing discovery that teams that are really efficient in a game, which would result in them performing much better than expected, caused their rating based on efficiency to improve.

Who would have thought that would happen?

🤷‍♀️


I mean some people might have realized that that is exactly the way that ratings systems like that are designed to work right from the start, but I guess not everyone.

It’s also worth noting that the scoring margin component of NET is capped at 10 points (1 point in OT). It’s the 5th component in the picture.

It’s the efficiency component that is influenced by game to game blowouts or not. Which is why I don’t think 10 points (to take an exaggerated number) here or there matters when total possessions is going to be so high.



Edit: Scoring margin also seems redundant and it’s really insignificant if capped at 10 so I almost wonder if it’s in there for casual fans who aren’t into possession efficiency.

 

[Vader_Storm]

It’s also worth noting that the scoring margin component of NET is capped at 10 points (1 point in OT). It’s the 5th component in the picture.

It’s the efficiency component that is influenced by game to game blowouts or not. Which is why I don’t think 10 points (to take an exaggerated number) here or there matters when total possessions is going to be so high.

View embedded media

Edit: Scoring margin also seems redundant and it’s really insignificant if capped at 10 so I almost wonder if it’s in there for casual fans who aren’t into possession efficiency.

What's important here is, and Ive said this before, no one knows exactly the "weights" used by the NET rankings. Each component has a particular weight to it. One metric may be weighted an 80 and another 5. Its vague. But its still quite similar to pomeroy and torvik and it moves in the exact same pattern as the latter two regardless of the 10 point cap it claims because it has efficiency metrics on offense and defense directly built into it.

 

[Vader_Storm]

So what you are saying is that you made the astonishing discovery that teams that are really efficient in a game, which would result in them performing much better than expected, caused their rating based on efficiency to improve.

Who would have thought that would happen?

🤷‍♀️


I mean some people might have realized that that is exactly the way that ratings systems like that are designed to work right from the start, but I guess not everyone.

I did not know until last season that the NET rankings moved in the exact same fashion almost to a T to both pomeroy and torvik. And Im sure the majority of people out there that follow far less basketball than me didn't know that either. I thought there were similarities. I did not think they were highly identical. They are not the same, but they move almost the same after game results.

 

[Pitt3Putt]

What's important here is, and Ive said this before, no one knows exactly the "weights" used by the NET rankings. Each component has a particular weight to it. One metric may be weighted an 80 and another 5. Its vague. But its still quite similar to pomeroy and torvik and it moves in the exact same pattern as the latter two regardless of the 10 point cap it claims.

Yeah I don’t know Torvik but I’m assuming that like KenPom it’s efficiency driven. And I think the NET largely is as well even without knowing the weighting.

 

[Joe the Panther Fan]

I did not know until last season that the NET rankings moved in the exact same fashion almost to a T to both pomeroy and torvik. And Im sure the majority of people out there that follow far less basketball than me didn't know that either. I thought there were similarities. I did not think they were highly identical. They are not the same, but they move almost the same after game results.

They are all measuring pretty much the same thing. Why wouldn't they all move pretty much the same way?

 

[Vader_Storm]

They are all measuring pretty much the same thing. Why wouldn't they all move pretty much the same way?

Torvik and Pomeroy are efficiency driven and scoring margin driven.


The NET has different elements like the link pittputt posted above. There is a big caveat here however. The NCAA has come out and said it has tweaked the NET system every single season since it came out with different weight adjustments and elements incorporated within in it. Not only do we not know the weights, we don't know how the weights have changed year by year. Its kind of silly to me. They should just leave it alone and run with it for a while instead of constantly tweaking it year by year. We'll see how similar it is to pomeroy and torvik this year. Last year is the first year where I really followed the NET pretty closely. I follow torvik closely every year.

 

[Joe the Panther Fan]

Torvik and Pomeroy are efficiency driven and scoring margin driven.

So is the NET. That's the whole point. It's been that way since day one. Heck, even SMF figured that out.

And yeah, they tweak it every year. And I agree, leave it alone for a couple seasons and see how it goes and then decide if you need to make changes or not. I would also be in favor of them being more transparent as to the exact formula they are using. I think the reason that they are not is that everyone knew the formula for the RPI, and everyone knew how to manipulate it (or at least attempt to manipulate it), and the NCAA didn't like that.

It's harder to manipulate a system that you don't know exactly how it works.

 

[Vader_Storm]

It's harder to manipulate a system that you don't know exactly how it works.

Run up the score,

 

[Joe the Panther Fan]

Run up the score,

Yeah, but if they really are limiting margin of victory (and who really knows if they are or to what extent that they are) then even that wouldn't really work.

Then you'd just get back to what I said the other day. Play better. If you keep playing better your NET will keep getting better.

 

[Vader_Storm]

Yeah, but if they really are limiting margin of victory (and who really knows if they are or to what extent that they are) then even that wouldn't really work.

Then you'd just get back to what I said the other day. Play better. If you keep playing better your NET will keep getting better.

It would still reflect in the offensive and defensive efficiency metric component of the NET. For all we know, that particular metric could have a weight of 95 to it. And if that's the case or something like it, then its going to move really in sync with the other 2 efficiency websites which is what happened last season. Without the weights its a guessing game. But Id still run up the score if it was me. And I think coaches are starting to catch on to this.

 

[Pitt3Putt]

Yeah, but if they really are limiting margin of victory (and who really knows if they are or to what extent that they are) then even that wouldn't really work.

Then you'd just get back to what I said the other day. Play better. If you keep playing better your NET will keep getting better.

I really think the low capped scoring margin piece is for casual fans who don’t know about possession efficiency and want to know if a 1 point win matters as much as winning by multiple scores.

The answer is no, but it’s not actually captured in that metric. It goes to the efficiency piece.

 

[Joe the Panther Fan]

It would still reflect in the offensive and defensive efficiency metric component of the NET. For all we know, that particular metric could have a weight of 95 to it. And if that's the case or something like it, then its going to move really in sync with the other 2 efficiency websites which is what happened last season. Without the weights its a guessing game. But Id still run up the score if it was me. And I think coaches are starting to catch on to this.

I really think the low capped scoring margin piece is for casual fans who don’t know about possession efficiency and want to know if a 1 point win matters as much as winning by multiple scores.

The answer is no, but it’s not actually captured in that metric. It goes to the efficiency piece.

Yeah, I think these two things go hand in hand. Like if they are limiting scoring margin, but scoring margin is not weighted highly, then they really aren't changing much by limiting scoring margin. And if you are limiting scoring margin but you aren't tinkering with individual game efficiencies, well, teams that blow someone out will be (by definition) more efficient than someone who wins a close game.

So limiting scoring margin could have an effect, but it is likely a really small one. Just be more efficient. Or in other words, play better. Your ranking will go up.

 

[Sean Miller Fan]

It’s essentially a very rudimentary gambling model. Meant to be predictive.

It does do some adjustments for scoring margin and other factors.

NET I don’t think does any of that. That said, over the course of probably 2,000+ possessions in a season don’t think 10 points in garbage time here and there matters that much. And if you are getting to garbage time possessions on a frequent enough basis to be a large portion of your total season possessions, you’re almost certainly blowing teams out or getting blown out enough to establish if you’re good or bad.

Our NET dropped 8-10 spots after Duke and we almost missed the tournament. That can't happen.

 

[Joe the Panther Fan]

Our NET dropped 8-10 spots after Duke and we almost missed the tournament. That can't happen.

Inexcusable beat downs like we suffered against Duke is what can't happen. Or at least not if you are on the bubble and you don't want to possibly miss the tournament.

 

[Sean Miller Fan]

Inexcusable beat downs like we suffered against Duke is what can't happen. Or at least not if you are on the bubble and you don't want to possibly miss the tournament.

Right but you cant drop a team 10 spots for 1 bad game against a really good team. And I know, if you get blown out against Duke, maybe you arent that good. Well, kinda, yea. But I believe in rewarding accomplishment not Vegas-style predictive measures.

 

[Joe the Panther Fan]

Right but you cant drop a team 10 spots for 1 bad game against a really good team. And I know, if you get blown out against Duke, maybe you arent that good. Well, kinda, yea. But I believe in rewarding accomplishment not Vegas-style predictive measures.

But the other thing that you can't wrap your mind around is that that game didn't happen in a vacuum. That was during tournament week, which means that almost all the teams around us in the rankings were also playing that day. And they all had results that affected the rankings. And at that point in a ranking system like that the difference between five or ten spots in the rankings is almost certainly very small. And when the differences between the teams are very small, and one of them is having an awful result while many of the rest of them are having "normal" results, well, the team with the awful result is going to drop spots.

What you seem to forget is that the exact sort of thing could happen in the RPI too, if you lost one game to a bad team. Like if we had played Louisville in last year's ACC tournament and we would have lost to them our RPI would have nose dived. Heck, our RPI would have gone down for merely playing Louisville last season. A loss to them would have been every bit a disaster as the Duke loss, even if the loss to Louisville was only by one point.