Talent Evaluation is Different in the NFL and NBA

The sudden emergence of Jeremy Lin has led people to wonder about talent evaluation in the NBA. Two recent examples — from Stephen Dubner in this forum and from Jonah Lehrer at Wired Science — both take similar approaches.  Both begin with the story of Lin, and then pivot to a discussion of the National Football League.  In essence, each writer argues that talent evaluation in basketball and football is similar.  

In my next two posts, I wish to address why I think talent evaluation in the NBA and the NFL is quite different.

(Photo: killthebird)

Let’s start with the NFL (my next post will focus primarily on the NBA). Brian Burke (of Advanced NFL Stats) and I have a chapter in The Economics of the National Football League: The State of the Art (Sports Economics, Management and Policy). In this chapter, we examine a variety of metrics designed to evaluate football players.  For the purpose of this discussion, I want to focus on the quarterback position (a position with an abundance of performance statistics).

In our discussion of NFL signal callers, we noted the percentage of a quarterback’s current season performance that could be explained by what the quarterback did last year.  Here is a sample of what we found for a variety of different performance measures:

  • Quarterback Rating: 15.0%
  • Completion Percentage: 31.1%
  • Passing Yards per Attempt: 22.1%
  • Touchdowns per Attempt: 10.1%
  • Interceptions per Attempt: 0.6%
  • Expected Points per Play:  21.0%
  • Win Probability Added per Play: 11.7%
  • Wins Produced per 100 Plays: 16.9%

To put these numbers in perspective, for many statistics in the NBA, season-to-season explanatory power exceeds 70 percent.  As one can see –relative to NBA players — NFL quarterbacks are quite inconsistent from season to season.  This is especially true for interceptions per play — a factor that has a big impact on outcomes — where explanatory power is below 1 percent.  In other words, just because a quarterback avoided interceptions in the past, it doesn’t mean the pattern will continue in the future. 

All of this tells us that even if you get to a see a quarterback play in the NFL, you are going to have a hard time knowing exactly what you will get from that same quarterback next season.  This is not surprising since much of what a quarterback does depends upon his offensive line, running backs, wide receivers, tight ends, play calling, opposing defenses, etc.  Given that many of these factors change from season to season, we should not be surprised that predicting the performance of veteran quarterbacks is difficult.

With this result in mind, let’s think about the draft.  Imagine a team decides it needs to select a quarterback this upcoming April.  A number of quarterbacks both excelled in college and are now available to enter the NFL.  At the top of the list are Andrew Luck and Robert Griffin III, the two players who led the Heisman vote. After these two, we see names like Ryan Tannehill, Brock Osweiler, Nick Foles, Brandon Weeden, Kirk Cousins, Kellen Moore, Russell Wilson, and Case Keenum (among others).  Of these, which player will be the best quarterback in the NFL?  

To sort through these players, the NFL holds a combine (which begins yesterday). This event will provide measures of each player’s height, weight, intelligence (i.e. Wonderlic score), and speed (40-yard dash time).  All of this is added to what the quarterback actually did on the college football field to determine which quarterback will be the “best.” 

So how much does all this information matter?  There are two issues to consider: 

  • How much do the people making the decision think this information matters?
  • How well does this information predict future performance? 

It was these two questions that Rob Simmons and I addressed in a paper published last year in the Journal of Productivity Analysis.  This paper looked at which factors explain where a quarterback is drafted, and furthermore, how those same factors relate to future NFL performance. 

When it comes to predicting draft position, we found that a quarterback’s height, his Body Mass Index, his Wonderlic score, his time in the 40-yard dash, whether or not he played in the top division of college football, and various on-field statistical measures are related to where a quarterback is drafted. 

So the information the NFL gathers does impact evaluations.  But when we turned to the second question, we found a result that may seem surprising.  None of the factors that were related to where a quarterback was taken in the draft predicted future NFL performance (and we considered a variety of per-play measures across a variety of time periods in a player’s career).   We did find that completion percentage in college — a factor that wasn’t related to where a quarterback was selected in the draft — did predict completion percentage in the NFL.  But our analysis indicated that less than 20 percent of completion percentage in the NFL can be explained by college completion percentage.

In sum, it is clear that people in the NFL do consider a variety of factors when choosing among NFL quarterbacks.  But the factors considered don’t predict what we see in the NFL.

Consequently, when we look at the NFL draft — with respect to quarterbacks — we should expect to be surprised.  Right now, people are convinced that Andrew Luck is a bit better than Robert Griffin III. And both of these players will be better than Ryan Tannehill.  And from what I can see from looking at various on-line draft evaluations, Tannehill is considered a better prospect than Russell Wilson and Case Keenum.

We should certainly expect that if Luck and Griffin III are taken in the first few picks of the draft, they will get to play more than those taken later.   In other words, draft position – as Rob and I noted in our paper – does impact playing time (just consider the number of chances Alex Smith of the San Francisco 49ers has gotten to play in his career).  But when we consider per-play performance (or when we control for the added playing time top picks receive), where a quarterback is drafted doesn’t seem to predict future performance.

And given what we see with respect to NFL veterans, this shouldn’t be surprising. It is difficult to predict the future for quarterbacks who have already played in NFL games.  Given that difficulty, we should not expect to have much success when we try and predict wh
at a player who has never played in the NFL will accomplish once they start playing on Sundays.

Fortunately for teams at the top of this year’s draft, how much they have to pay these prospects has declined considerably.  Teams no longer have to give large guaranteed contracts to players like JaMarcus Russell, Matt Leinart, David Carr, Joey Harrington, Tim Couch, Akili Smith, and Ryan Leaf.   Each of these quarterbacks was highly touted on draft day.  And each disappointed the people who gave them many millions of dollars to try and play in the NFL. 

With the new Collective Bargaining Agreement, top draft picks will now receive far less than what they were paid in the past. So when teams miss on these quarterbacks – and given what we have seen in the past, we can predict with confidence that teams will miss on some of these players – the cost is going to be much lower.

Okay, let’s now pivot back to the NBA.  We have just seen that predicting performance in the NFL is difficult.  And that confirms part of the story told by Dubner and Lehrer.  But does that really help us understand Jeremy Lin?  In my next post, I am going to discuss what the published research tells us about player evaluation in the NBA.  And that story is going to demonstrate that the NBA is not really the same as the NFL.


I agree with everthing you said, but a rookie QB's future success is highly dependant upon their Offensive Line...mainly (for protection and lasting longer than a season - see Tim Couch), and coaching/offensive schemes. Yet another layer of factors that can determine success and failure of young rookie QB in the NFL.


I wonder what happens when you use # of sacks/hurry-ups to stats like QB Rating or Interceptions. Definitely seems like these would be linked.


I work in athletics, specifically college recruiting. I am finding a disproportionate number of athletes on my list with last names that are from the beginning of the alphabet. I have read one of your books recently and am interested to know if you have done research on the athletic success based on a persons last name. I have ideas on why this might be the case, but wonder if you have any information to share?



I think in the NFL, there are a lot more variables that impact both the evaluation of a player coming out of college (quality of both his teammates and the opposition he played against, for example) than in the NBA, where star players are much more dominant by virtue that they generally play around 18% of their teams overall minutes.

Likewise, once a player reaches the NFL, there are also a huge number of variables in terms of both the quality of his own teammates, his coaching, as well as the quality of the opposition he faces (which changes quite a bit from year to year given the NFL's scheduling formula). And there's a lot more player movement from team to team via free agency in the NFL than there is in the NBA as well.

Take Alex Smith (whom you already mentioned), for example. He's had seven different offensive coordinators in his first seven seasons. Likewise his performance in 2012 is likely to be worse than in 2011 simply because the 49ers play a much tougher schedule (including at Green Bay, at New Orleans, at New England, at NY Jets) in 2012 than they did in 2011. And every single member of the 49ers offensive line has changed during his seven years in SF as well. A lot of factors that are out of his (or any QB's control)....



Thought this was up your alley

Selma Casey

Dave, I look forward to reading your chapter in "The Economics of the National Football League."

Unfortunately for me, that book is listed at $135 and the lowest online price I can find for it is still over $100.

I tried to take a closer look at your chapter at Springer's publisher website (http://bit.ly/Ab4kFD) but they want $25 to let me take a look.

So this poor student is, I fear, out of luck as far as trying to understand the point you're making.

Would you please tell us if this paper has been published in a more easily-accessible journal? If so, which one?

David Berri

I have never studied last names and athletic achievement. I once noted that Matt Millen had a problem with the Lions because every head coach he hired had a last name that started with "M" (just like Matt Millen).

Such is the nature of academic collections. They are quite expensive and generally purchased by libraries. Perhaps you can have your library order a copy (or use inter-library loan).


The seeming randomness of INT stats reflects the fact that an INT is only part of the time attributable to the quarterback, but is often due to the receiver. Balls bounce off receivers' hands into the hands of a DB -- no fault of the QB. Even more important, QBs often throw intentionally not to the receiver but to a "spot", with the ball in the air before the receiver arrives there. If the receiver runs an incorrect route, then only a DB is on the "spot". Other times, the QB and the receiver do not have the same interpretation of a required situational adjustment in the designed route, so the QB throws at the wrong time or to the wrong spot. That INTs are seemingly random is thus unsurprising.


" where a quarterback is drafted doesn’t seem to predict future performance."

Dave, with all due respect, do you have a response to the criticisms of your methods (particularly omitting late drafted players who never play) in making this finding:




I'm glad you linked to those articles, because both of them (and many, many others) point to the key problem with Berri's argument. If you point simply to rate statistics and fail to account for both sample size and causation, you're going to run into trouble. Or to borrow a hypothetical example from one commenter, it's possible that the average 7th round pick who starts at QB in the NFL for ten seasons is just as good as the average 1st rounder who starts for ten seasons. Does that imply that there isn't any difference between 1st rounders and 7th rounders? Not if the 1st rounders were 5 or 10 or 50 times more likely to play 10 years in the first place.

Berri seems to dismiss the differences in playing time by implying that the guys who didn't play were just as capable of success as the guys who played a lot. This, in turn, leads to a number of troublesome conclusions:

- Playing time doesn't indicate aptitude for success.
- Coaches are incapable of recognizing who gives their team the best chance to win and playing them the most.
- Rate stats matter more than aggregation. In other words, a QB who completes 3 of 5 passes in his career has been as successful as a QB who completes 3,000 out of 5,000 passes.

I've seen studies by Football Prospectus, ESPN Stats, and Sports Illustrated that all pointed to the fact that well over half of elite "franchise quarterbacks" have been drafted in the first round. No, this doesn't mean that all 1st rounders succeed - not even close. But to imply that there is no discernible difference in quality between top QB picks and late picks is inaccurate and misleading.