For several years I have often thought about how great it would be if I could play in a fantasy league that included an advanced defensive metric as a category. Fantasy largely neglects an entire aspect of baseball. The conventional categories’ connections to defense are indirect and loose. I have heard of some leagues playing with Assists, Putouts, and/or Errors, but I think we can all agree these have their shortcomings. There are better options available now.
Range-Based Defensive Metrics
Outs Above Average (OAA) is the most recent (publicly available) attempt by the analytic community to capture a player’s defensive skills. It is a range-based metric put out by Statcast in 2017. Prior to 2020, it was an outfield-only metric, but it appears to have been calculated retroactively for all players going back to 2017. A positive number is above average, a negative number below-average. It is cumulative so a great defensive player that got a lot of chances would have a large number. Full season numbers generally range from 30 to -30. The Statcast description indicates that rational numbers are possible, but their leaderboard only gives whole numbers currently.
In 2003 Ultimate Zone Rating (UZR) was introduced, and it has a similar foundation based on range. In this time of rapidly developing advanced metrics, it seems like a dinosaur, but it is still quite respectable. The biggest difference between it and OAA is that UZR generally does not take shifted plays into account. Thanks to Statcast, OAA knows exactly where a player was positioned before a ball was hit. UZR does not. With the ever-increasing use of shifts, this difference is likely to become even more pronounced. OAA also takes a hitter’s sprint speed into account—fielding a grounder hit by Albert Pujols requires less urgency than one hit by Tim Locastro. UZR doesn’t do that.
ERA Estimators and Defensive Metrics
This may seem like a bit of a shift, but I began to wonder how defensive metrics could account for differences between various ERA Estimators and ERA. Fortunately, this also serves as a way to test out the validity of these defensive metrics.
FIP, xFIP, SIERA, and other such ERA Estimators attempt to remove what a pitcher can’t control. The quality of the defense behind him is a big one. A team that had a significantly lower actual ERA than expected ERA may have had an excellent defense, and vice versa. Looking at how much of the variability of this difference is explained by various team defensive metrics seemed like the logical next step. Plus, I’m beginning to suspect that determining R and R2 values is all I know how to do statistically.
I started by gathering FIP, xFIP, and SIERA numbers by team for each year of the OAA-era, 2017-2020. I looked for team numbers for xERA and pCRA, but couldn’t find them. Ultimately, there certainly are differences between all of these ERA Estimators, but I don’t think including xERA, pCRA, or others would have made much of a difference here. I then calculated the R2s for each team’s OAA, UZR, and Fielding % with each of those ERA Estimators. I am aware of the fundamental problems with using Fielding % as only earned runs are included in these Estimators, but Team Fielding % has been floated around in the past as a measure of the overall quality of their defense. I also thought it was important to include a traditional defensive stat.
| OAA-R2 | UZR-R2 | F%-R2 | |
|---|---|---|---|
| FIP-ERA | 0.227 | 0.134 | 0.002 |
| SIERA-ERA | 0.230 | 0.207 | 0.001 |
| xFIP-ERA | 0.168 | 0.191 | 0 |
| Mean | 0.208 | 0.177 | 0.001 |
Note: OAA and UZR all have p-values below .05. All p-values for Fielding % were close to 1
Most unsurprising is how utterly useless Team Fielding % is when trying to explain Estimator-ERA differences. On the other hand, team OAA and UZR can explain between 13% and 23% of the variance. The other 80% or so can be explained by imperfections with the Estimators, OAA, and UZR. Luck will always play a part too. While not great, R2s of around .20 in baseball research are respectable.
I interpret this as meaning that OAA and UZR measure something that is important, that has a real impact on the game. I am not particularly interested in Fantasy categories that have little influence on Big League outcomes.
Everyone has their favorite ERA Estimators, but FIP seems to be the most versatile and reliable, with SIERA coming in second. I would put the most stock in those R2s.
Based on this, OAA has a small, but distinguishable advantage over UZR. Let’s look at it from another direction now.
Defensive Metric Year-Over-Year Stickiness
In order for a stat to be included as a Fantasy category, it should be at least somewhat predictable. A completely random category would lead to many frustrated managers, and, in all likelihood, human extinction. In order for one of these defensive metrics to be considered, it needs to correlate with itself year-over-year. A decent correlation would suggest that said metric is measuring actual skill.
Obviously, this must be done on a player level. I included players that had a minimum of 50 opportunities in each season from 2017-2020. This resulted in 136 players being selected.
| OAA | UZR | F% | |
|---|---|---|---|
| YoY | 0.42 | 0.37 | 0.50 |
| 2 Year | 0.42 | 0.38 | 0.55 |
| 3 Year | 0.37 | 0.37 | 0.52 |
| Mean | 0.40 | 0.37 | 0.52 |
A sign of a good table is that it needs no explanation… The YoY row shows how well one year of data correlates with the next (e.g. 2017 to 2018, 2018 to 2019). 2 Year shows how well two consecutive seasons of data correlate with the third year (e.g. 2017-2018 to 2019, 2018-2019 to 2020). 3 Year shows how well 2017-2019 correlates to 2020. The above table shows means of YoY and 2 Year correlations.
I chose to display the results of the correlation in this manner to show how predictable these metrics are with increasing sample size. Future work could determine stabilization rates. I’m guessing the short 2020 season befouled the results somewhat, especially the 3 Year. However, they are all on the edge of what is considered a moderately strong correlation. I anticipated Fielding % to be pretty good. It is a measure of a player’s hands and throwing accuracy and one would expect those skills to remain stable, even with age. However, given the results earlier for Fielding %, I would still consider it inappropriate for fantasy purposes. It is interesting to see, though.
The correlation differences between OAA and UZR are negligible here. However, previous work by Craig Edwards at FanGraphs in 2018 came out with stronger results in favor of OAA. Their correlations for YoY OAA were closer to .70. This puts it into better company with other metrics. Their work was done on a rate-basis, so I would expect their correlations to be stronger than mine.
Final Thoughts
I love the idea of creating an entirely new dimension of Fantasy Baseball. Adding an advanced defensive metric creates something that more closely mimics the game played on the green grass. Previously fringe, glove-first players suddenly vault into the realm of interest. Trailing in OAA towards the end of a Head-to-Head? Add Nick Ahmed. Move Nick Castellanos to the DH or bench spot.
My favorite plays to watch are infielders covering large distances to get to a ball and then making seemingly inhuman throws. This element should be included in fantasy.
Add it as a 6th non-Pitcher category. Maybe use it to replace that pesky SB category. Based on the numbers I have seen, Outs Above Average seems like the best candidate to break down the metaphoric defensive door.
Considering the rapid growth of the analytical community, it will likely be replaced by something better in the future, but we have something good now. Why not add a third dimension?
Photos by All-Pro Reels/Flickr, Andy Lewis & Juan DeLeon/Icon Sportswire | Adapted by Justin Redler (@reldernitsuj)