Many fans may have hated when David Letterman left TV but, if you ask him, he’ll tell you he actually overstayed his welcome. Sounds harsh but he has a pretty good reason for feeling that way.

“Here’s the mistake I made — I stayed on television way too long,” the 71-year-old told Ellen DeGeneres, who totally dismissed the statement on The Ellen DeGeneres Show. “Yes, it’s true,” he responded.

“I’ll tell you what happened: It turns out, nobody had the guts to fire me. I should have left like, 10 years ago. You want to make sure you have some energy to direct toward other things. Now you — nothing but energy — are doing other things while you’re on television, so that’s great. I did not.”

David, who started hosting in 1982 on Late Night with David Letterman on NBC, stayed on the air until 2015 when he retired from The Late Show with David Letterman on CBS. He even had a daytime talk show in 1980 titled The David Letterman Show. Over the years, David’s shows have garnered 16 Emmy wins and 112 Emmy nominations, per Variety.

“All I cared about was myself and then the show was gone so I had to realize, ‘Oh, I’ve been looking through the wrong end of the telescope.’ There is more to life than, ‘So tell me about your pet beaver,'” he added.

In the past, David told Closer Weekly he missed nothing from hosting except for the live music from Paul Shaffer. During their decades-long partnership, the two only had one disagreement and, spoiler alert, it was barely a thing. The pair now work together on the Emmy-nominated Netflix series My Next Guest Needs No Introduction with David Letterman.

This sentiment comes soon after a similar one made by one of David’s late-night rivals, Jay Leno, said hosting The Tonight Show felt like 100 years ago and that he doesn’t really miss it.