June 16, 2009
Up and Coming
Refining The Estimate

by Iain Fyffe

Last week in Up And Coming we examined the results of some initial attempts to hammer raw statistical data into a usable estimate of a draft-eligible player's future hockey career. This week we realign our sights a bit, and refine our estimate a great deal.

One change I needed to make was the target that we’re trying to estimate. Previously I had tried to estimate the player’s Career score. This was problematic because of injuries. A good player who misses a lot of time due to injury will have a low Career score, due to the missed time. At this point, we don’t have any way to predict future injury, so injury should not be allowed to affect our rankings. Therefore the new target to be estimated is the player’s Career score per game played. Using a rate stat allows us to avoid the injury problem.

In the future, if we are able to develop methods of predicting career time lost to injuries, we will build that into our forecasts. However, at this point, we can’t say that we should have known Eric Lindros would have concussion issues, or that Pat Peake would suffer his myriad of injuries.

The development of this estimate was more rigorous than before. It began with a linear regression using goals per game, assists per game, and penalties per game to produce an initial estimate. From that point, adjustments were added for the items discussed in last week’s column (age, goal/assist balance, penalties), though often to different degrees. For instance, the adjustment I had made previously for players born in September to December was too heavy-handed. This became obvious when I applied the formula to the 2009 draft class, and John Tavares came out ranked fourth among the OHL forwards, which is obviously wrong. This also cemented the need for a more rigorous method of developing the estimate. Adjustments were only made to the degree that they improved the accuracy of the estimate.

There was at least one interesting finding from the regression analysis itself. I used goals per game and assists per game separately in the regression, rather than just using points per game, which would have assumed that goals and assists are equally valuable in predicting future success. As it turns out, with respect to the age-17 season, goals and assists are precisely equal in value. However, in a player’s age-16 season (if he played in major junior), his goals scored are a much more important indicator of success than his assists.

I won’t divulge the specific formula, partly due to its complexity and partly because further refinements are likely, but we can look at some of the results to get an idea of the robustness of this estimate. These are the top 50 players in our data set, according to the estimate. Each player’s actual Career/GP is provided for comparison.

				Career/GP	Estimate
1.	Lindros, Eric		1.11		1.00
2.	Thornton, Joe		1.14		 .99
3.	Gratton, Chris		 .75		 .79
4.	Savard, Marc		 .76		 .76
5.	Ricci, Mike		 .74		 .71
6.	Legwand, David		 .81		 .68
7.	Primeau, Keith		 .78		 .66
8.	Roche, David		 .35		 .66
9.	Nolan, Owen		 .88		 .63
10.	Allison, Jason		 .87		 .63
11.	Cleary, Daniel		 .59		 .63
12.	Tkaczuk, Daniel		 .42		 .62
13.	O'Neill, Jeff		 .77		 .62
14.	Wren, Bob		 .53		 .62
15.	Stillman, Cory		 .68		 .61
16.	Arnott, Jason		 .89		 .58
17.	Kilger, Chad		 .49		 .58
18.	Bowler, Bill		 .52		 .58
19.	Bonsignore, Jason	 .38		 .57
20.	Moreau, Ethan		 .49		 .55
21.	Warriner, Todd		 .46		 .55
22.	Bell, Mark		 .57		 .55
23.	Sim, Jonathan		 .49		 .53
24.	Harvey, Todd		 .54		 .51
25.	Taylor, Chris		 .46		 .49
26.	Simon, Chris		 .56		 .49
27.	Young, Jason		 .43		 .48
28.	Convery, Brandon	 .49		 .48
29.	Cheechoo, Jonathan	 .71		 .47
30.	Seguin, Brett		 .41		 .47
31.	Zehr, Jeff		 .29		 .47
32.	Miller, Colin		 .32		 .46
33.	Intranuovo, Ralph	 .52		 .46
34.	Park, Richard		 .46		 .46
35.	Volchkov, Alexandre	 .18		 .46
36.	Milley, Norm		 .42		 .45
37.	Gauvreau, Brent	         .31	         .45
38.	Papineau, Justin	 .45		 .45
39.	Peake, Pat		 .54		 .45
40.	Sarno, Peter		 .49		 .45
41.	Fata, Rico		 .51		 .43
42.	Stevenson, Shayne	 .36		 .43
43.	May, Brad		 .50		 .43
44.	Bowen, Curtis		 .24		 .42
45.	Primeau, Wayne           .40             .42
46.	Avery, Sean		 .57		 .42
47.	DeFazio, Jerrett	 .18	         .42
48.	Jinman, Lee		 .37		 .41
49.	Winch, Jason		 .33		 .41
50.	Barney, Scott		 .38		 .41

Overall the results look very good. We still have some significant false positives here, of course. David Roche is one of the biggest, and the scouts certainly beat the estimate with him. The next miss on the list is Daniel Tkaczuk, but of course he fooled the scouts too, being drafted 6th overall in 1997. Going case-by-case is no way to determine accuracy, of course. There will always be times that the scouts do better than the numbers, and others when the numbers are better. Marc Savard, whose Career/GP is very high, and estimated very accurately, was drafted 91st overall in 1995. The crucial question is: who gets it right more often?

As a quick and dirty check for accuracy, I calculated the coefficient of correlation between the Career/GP scores and the estimate. The result is 0.7, which is a very strong correlation. The correlation between the players’ draft positions (with undrafted players considered to have a position of 500) and Career/GP is -0.5, which is pretty good (a negative correlation here is good, because a high draft position is a low number). But it seems we’re outpredicting NHL teams here, by a fair margin.

Ultimately, of course, the purpose of this exercise is not to look to the past, but to the future. What does this formula predict for the current crop of OHL draft-eligible forwards? What does it predict for John Tavares?

Let’s hide Tavares’ estimate for now, and look at the others a bit.

				CSB rank	Estimate
1.	Tavares, John		1		.XX
2.	Kadri, Nazem		4		.55
3.	Duchene, Matt	        2		.53
4.	Kassian, Zack		5		.47
5.	Werek, Ethan		8		.43
6.	Hutchings, Alex		9		.41
7.	Holland, Peter		3		.40
8.	O'Reilly, Ryan		6		.40
9.	Latta, Michael		7		.39
10.	Beck, Taylor		10		.38
11.	Szwarz, Jordan		11		.35
12.	Varone, Philip		14		.35
13.	Wilson, Garrett		12		.34
14.	Fine, Michael		16		.32
15.	Agozzino, Andrew	N/A		.31
16.	Foligno, Marcus		17		.31
17.	Mayer, Jordan		18		.31
18.	Randell, Tyler		15		.30
19.	DeSousa, Chris		N/A		.29
20.	Erlich, Daniel		N/A		.28
21.	Cizikas, Casey		19		.28
22.	Sisca, Matt		N/A		.28
23.	Clifford, Kyle		23		.25
24.	Carnevale, Taylor	N/A		.25
25.	Tipoff, Matthew	        22		.23
26.	Judson, Luke		20		.22
27.	Andreoff, Andy		24		.20
28.	Payerl, Adam		21		.17
29.	Tatrn, Zach		13		.10

On the whole, Central Scouting does quite a good job ranking the players. For the most part, the differences are just a few places in the rankings, which is to be expected. There’s enough uncertainty in the estimate that such small differences in the rankings can’t be said to mean anything. The only really big miss is Zach Tatrn, who the scouts like but the numbers predict a marginal minor-league career.

Other than Tavares, none of these players forecast to be exceptional NHL players. Nazem Kadri is the best after Tavares, and his estimate of .55 puts him in a class with Ethan Moreau, Todd Warriner and Mark Bell, meaning he’ll likely have a solid, but unspectacular NHL career.

So what about Tavares? The estimate pegs him at .99. That’s a huge number. In our data set, Eric Lindros has the best estimate at 1.00, and Joe Thornton is next at .99. No one else is above .80. So once again, we see that John Tavares is in an elite class of prospects. He’s an A+. This is further evidence that he should become a franchise-type player.

Lately I’ve been wavering in my support for Tavares to go ahead of Victor Hedman in the draft. I was concerned due to his age and his lack of increased production through his major-junior career. This analysis alleviates those worries. Hedman may well still deserve to rate ahead of Tavares, but this demonstrates he has very tough competition in Tavares. When I finally develop a method to forecast Swedish defensemen, we’ll see what the numbers say. However, that certainly won’t happen before June 26, and by that time it will be too late. The Islanders will have made their decision, but looking ahead to the 2010 draft, we should have some very good estimates on the eligible players. Will it be Vladimir Tarasenko, Taylor Hall, or perhaps someone else ranked first? Only time will tell.