Sports fans will probably be aware that Roger Clemens is currently before Congress, arguing that the Mitchell Report wrongly tagged him as having used performance-enhancing drugs. And last week, his agents released the “Clemens Report,” arguing that his career statistics somehow exonerate him. The full marketing spin is available here.
I was interested in understanding how they could “prove” his innocence by crunching numbers, and in an effort to make sense of it all, I sat down in a Wharton conference room with three fellow data hounds – Eric Bradlow, Shane Jensen, and Adi Wyner. With two stats professors and a stats/marketing prof in the room, I felt a bit outmatched. But it sure was fun to work through the issues together.
The main argument in the Clemens Report is that there is nothing unusual in his career numbers, and, in fact, his performance is quite similar to that of Nolan Ryan. But we note the following:
[S]uch comparisons tell an incomplete story. By comparing Clemens only to those who were successful in the second act of their careers, rather than to all pitchers who had a similarly successful first act, the report artificially minimizes the chances that Clemens will look unusual.
There’s a pretty neat trick at work here: if you compare Clemens only to those who had a terrific last decade of their careers, then the last decade of Clemens’ career doesn’t look that unusual. To sidestep this, we suggest that “[a] better approach to this problem involves comparing the career trajectories of all highly durable starting pitchers.”
So we put together data on all 31 other pitchers since 1968 who started at least 10 games in at least 15 seasons and have pitched at least 3,000 innings. This broader comparison group yields some pretty different conclusions than the Clemens v. Ryan contrasts.
A picture is worth a thousand words, and here we show simple quadratic fits to the data for Clemens v. controls:
The Clemens Report is also notable for its near-exclusive focus on his ERA. Now, any Sabermetrician will tell you that this is not a particularly reliable statistic, and that it bounces around a lot more than a pitcher’s true performance. This is a problem because noisy data can obscure an underlying pattern. So we supplemented our analysis by examining a range of alternative indicators, including walks and hits per inning pitched (see right panel, above).
We conclude that “the available data on Clemens’s career strongly hint that some unusual factors may have been at play in producing his excellent late-career statistics.”
To be clear, we don’t know whether Roger Clemens took steroids or not. But to argue that somehow the statistical record proves that he didn’t is simply dishonest, incompetent, or both. If anything, the very same data presented in the report – if analyzed properly – tends to suggest an unusual reversal of fortune for Clemens at around age 36 or 37, which is when the Mitchell Report suggests that, well, something funny was going on.
You can read our full analysis in today’s Times, here.
UPDATE: Roger Clemens’ crisis management consultants have just released a rejoinder to our analysis, available here. Further coverage: Lester Munson at ESPN.com, a less flattering analysis at MLB.com, and another piece at ESPN.com (leading to hundreds of comments).
and 
. You can also watch it on 
Since you refer to a comparison between Nolan Ryan and Roger Clemens, your graph should plot Nolan Ryan’s performance (in addition to the control group).
Professor Wolfers, a few comments:
1) I believe there could be an error in your culling formulae. I find 34 pitchers with 15 or more seasons since 1968 who have at least 3,000 innings pitched in that timespan. Here’s my list: http://sturgeongeneral.wordpress.com/files/2008/02/wolfers_list.pdf Who don’t you include from that list, and why?
2) The list of comparables is interesting, but not particularly useful for making a very specific conclusion about the shape of Clemens’ performance. Any pitcher who pitches 3,000 innings over a 15+ season career is in the far right tail of the distribution of major league pitchers, who themselves are in the far right tail of pitching talent across the world. You can find interesting things about each guy in your list, but the sample is just so small that drawing broad conclusions is very dangerous. As Cyril says, do you think Warren Spahn was on steroids?
3) Clemens had the fourth highest (unadjusted) strikeout rate of the group I found (behind Randy Johnson, Nolan Ryan, and Curt Schilling). Right behind him are John Smoltz, Chuck Finley, Steve Carlton, Mike Mussina, and Tom Seaver. As you may know, high-strikeout pitchers tend to age better (show less of a dropoff in their performance) than do low-strikeout pitchers.
To test this, I broke the group of 33 (excluding Clemens) into four quartiles, and built a multiple regression model to predict WHIP by age and quartile. Here’s a picture of the output: http://sturgeongeneral.files.wordpress.com/2008/02/wolfers-chart.png
Each curve looks much like the output of your model in the NYT article, which means I must at least have partially done something right
. However, if you’ll notice, quartile one pitchers (of which Clemens would have been one, had he not been excluded) have the shallowest aging curve – they don’t tend to increase their WHIP as much as they get older. This is exactly what we’d expect.
Clemens’ career certainly has an unusual shape, and I myself believe he took steroids to achieve it. But I don’t believe your analysis is particularly probative of that conclusion.
I appreciate the in depth analysis, but it is not deep enough for conclusions. The following factors still need to be factored in: who was catching (catchers usually call the pitches, better catchers do better calling pitches); team, Clemens was on better teams later, meaning the better offensive players were playing with him, not against him.
Showing the lines of Clemens against 4 or 5 of the 31 would give an idea of true trajectory of the sample. Others could have similar paths, and Ryan must be close (you did not show the one they think is most comparable, and he among the few pitchers that do compare).
So how does the graph of Nolan Ryan versus the other pitchers look? If his team is suggesting that he is normal within the pool of unusually long-playing pitchers, it is a valid argument to make: special, but not so special as to be singularly unique and thus, suspicious.
What are the curves like for the other successful late-career pitchers, such as Johnson, Schilling, and Ryan? Comparing their curves to Clemens would be useful, as it would show if the path he took to his late-career success is an anomaly. While it certainly is compared to most pitchers with early-career success, the more relevant question it seems is if the path he took is different than that taken by similarly successful late-career pitchers, like those mentioned before.
Additionally, do you control for other pitchers accused of using steroids? Charting their career paths and comparing them to Clemens’s may be useful, as well.
Here’s what I get when I filter to only include seasons of 90+ IP.
Schilling and Johnson have more typical aging curves, but Ryan’s is even more of an outlier than is Clemens’.
http://sturgeongeneral.files.wordpress.com/2008/02/aging_curves.png
Wouldn’t it make more sense to do a seperate charting of how “performance enhanced” pitchers performed with age. If Clemens’s trend matches more evenly with a juiced pitcher, than a typical “all-star” the trending with age would show it, especially in light of your ERA curve.
I’m not sure if comparing him to pitchers from 1968 is accurate. There have been advancements in exercise science in general that would give him an advantage over pitchers from the 60s.