As fall practice begins, it’s getting closer and closer to prediction time for all of your favorite writers. And for Notre Dame in 2017, that means that every one-line prognostication will devote a few words to Brian Kelly’s future. “Notre Dame: 9-3 with an improved defense under Mike Elko, cooling Brian Kelly’s hot seat.” Or conversely, “Notre Dame: 7-5, with several close losses leading to Brian Kelly’s dismissal.”

A favorite message board topic is also how many wins Kelly needs to keep his job heading into 2018. Despite Jack Swarbrick’s public statements of confidence in BK, I think there’s widespread agreement that six wins or fewer this fall will lead to a transition. Similarly, improvement from 4-8 to at least 9 wins or more seems destined for Kelly to return. Seven or eight wins is a grey area, and then it starts to matter more about how the wins and losses take place and the margins of victory or defeat.

These predictions and debates bring up some larger issues – how should Jack Swarbrick and others be judging football performance year to year? What’s the standard, and how should wins and losses figure in along with other measures of program quality? Let’s tackle those issues first, and then look ahead to what needs to be seen in 2017 for Kelly to return for another season.

Where should the bar be set for a Notre Dame coach?

Ultimately, every coach will be judged based on wins and losses. A team will never make the College Football playoff with less than 10 wins, and even a 10-win playoff team is unlikely unless there’s 2007-level chaos again. As Herm Edwards said, “You play to win the game!” and there’s no advanced statistics that will save a coach who is consistently losing more games than an athletic director and fanbase expects.

Simultaneously, there needs to be some recognition that wins are fickle and often the result of some year to year randomness that you’ll inherently deal with in a small sample size like 12-13 games. Some years your schedule will break in the right direction and random events like fumbles recovered and deflected passes turning into interceptions will bounce your way. Other years will be like 2016, with a number of losses in close games that would be difficult to replicate even if that was your goal.

The details beyond wins matter, which is how a 10-win season like Notre Dame in 2015 is argued as either one of Brian Kelly’s finer coaching performances (10-wins despite tons of injuries and playoff contention) or a disappointing measure of the program’s current ceiling (can’t beat top teams, clearly undermanned against a championship caliber program like Ohio State). West Virginia and Michigan both had 10-win seasons last year in P5 conferences, but the degree of difficulty and margin of victory matter, especially if you’re looking to project performance and progress moving forward.

Other sports have widely used advanced stats to determine if teams are over and under-performing relative to their records – Pythagorean expectations have been used for years, starting in baseball but recently applied to basketball and pro football.  These stats, along with Bill Connelly’s second-order wins, have been validated in identifying teams over and underachieving relatively to their wins and losses and their predictive value in then leading to regression toward a “truer” record in the following seasons.

If the goal is setting a standard for performance year to year, then program quality is a steady year to year barometer to complement wins and losses. Program quality would be like your baseline temperature, with wins (and their inherent randomness) acting almost like windchill or humidity that due to randomness and small samples can deviate from expected outcome. In this analogy, last season for the Irish would be like a 58-degree day with 40 mph gusts that pull the “feels like” temperature down to 32. But keeping an eye on these circumstances and program quality could prevent a hasty decision based on W-L record – either in investing further in a coach who is overachieving and benefiting from some luck that’s unlikely to continue, or firing a coach that’s been the victim of a series of unfortunate events .

For Notre Dame, fans could debate where the bar should currently be set for all eternity – playoff contention every year? Making the playoff once every other year? Double digit wins in the vast majority of seasons with a couple free passes for a proven winner? There will never be unanimous agreement, but I’ll propose a starting point that’s maybe conservative but realistic and attainable – top-10 quality almost every year. Over the past seven years (consistent with Kelly’s tenure), Stanford and Oklahoma seem like what Notre Dame should be expecting as they’ve established themselves as consistently strong programs with close to the same talent coming in via recruiting.

A top-10 quality team should almost always guarantee the record needed to consider it a successful season – 63 of the 70 teams finishing in the top 10 of F+ over the last seven seasons won ten games or more. The inverse, however, holds true less often – there are multiple examples each season of power five schools with double digit wins whose quality (per F+) was only that of a top 20-30 team or worse, like 2015 “playoff contender” Iowa (38th in F+). Now if there’s a consistent trend of under-performance in win-loss record compared to program quality, it could be symptomatic of a larger issue with a coach and program. Which leads us to…

How has Brian Kelly performed against the standard of a top-10 team?

Utilizing F+, a combination of Bill Connelly’s S&P+ and Brian Fremeau’s FEI ratings, both of which are strong advanced stats systems incorporating things like opponent adjustments and throwing out garbage time, it’s interesting to look at Notre Dame under Kelly. He inherited a team from Charlie Weis that finished 25th in S&P+ in 2009, and brought consistent improvement over the next three years. Kelly reached a top-10 level by year two, and then repeated it in year three with some good fortune resulting in a title game appearance.

The trend line after 2012, however, isn’t pretty. Looking at winning percentage in these seasons makes the decrease in quality look less dramatic, as the 2010 and 2011 teams drastically under performed relative to their quality, going 8-5 with a top-15 and then top-10 level teams. It’s a huge concern Kelly’s teams haven’t performed at a top-20 level (much less top-10) in years four, five, and seven. And probably the only thing that’s saved Kelly’s job is the equity he built with Jack Swarbrick from his work in years 1-3, combined with some leading indicators like recruiting remaining solid.

The graph above also begs the question of why there are at least three years (2010, 2011, 2016) in the BK era where the team’s record has pretty dramatically under performed relative to program quality and expected wins. Under performing relative to program quality is a secondary problem – the biggest concern should be a team that hasn’t performed at the quality level it should. As Bill Connelly wrote when explaining differences in second order wins among coaches who consistently over and under-achieve, “this is only one aspect of good coaching … creating a good team and winning easy games is a larger aspect.  But if it’s happening consistently, it erodes any margin of error, and raises the question of whether the coaching staff’s preparation, decision-making, or conditioning are causing losses in close games.

Where should the bar be set in 2017?

After finishing 29th in F+ last season, combined with performances below the top-25 level in three of the past four years, Kelly needs to show strong improvement to argue he’s made changes that are correcting the program’s course back to a top 10 (or near) level. A top 10-15 performance would represent a step in the right direction, and unless lightning strikes twice, will lead to enough wins that Kelly should be back. Any worse than top-15 in program quality, and that’s where lucky with wins and losses will determine Kelly’s fate. His failures over the past few seasons give BK less wiggle-room, so even with solid improvement in program quality, a second consecutive season with poor performance in close games, bad turnover luck, and other factors could lead to his dismissal.

Both S&P+ and FEI agree in the preseason that Notre Dame’s most likely record is 8-4. Both are assuming the Irish improve – S&P+ has ND 17th, FEI 19th – but will that be enough if it holds true? Close game luck is due for some regression to the mean, but what if Notre Dame again plays too many close games, where the expected winning percentage should still hover around .500 year to year? It’s easy to envision scenarios where an Irish team that’s around the 15-20th best team nationally could win 10 games if the schedule turns out a bit softer than it looks now. Just as easily could the Irish drop a low-possession game with Navy, lose close contests with quality opponents, and finish 6-6.

Strangely it seems like Swarbrick’s decision next offseason will come to a few bounces of fortune either way – another underachieving season spells the end of the BK era. A bounce-back probably ensures at least a couple more years of Kelly at the helm, but could simultaneously not really represent a big step forward in program quality. As a fan I’ll be selfishly hoping for clarity one way or another. I’m looking for either a huge comeback both in program quality and record that’s truly indicative of improvement that will be sustained and lead back to consistent top 10 performances in the future (which seems less likely, but more fun to root for and I’m an optimist) or another season that makes it abundantly clear a new coach is needed to meet the standard of success for the program (which would be terrible, but better long-term for the program).