Johnny Carson was a prick to many people as well as a perfectionist with his show. He cut off Joan Rivers not because she tried to start her own show on FOX in 1986 but because Rivers never gave Carson a heads up about it beforehand. He found out about it watching TV.
Carson also held more power in television than any on-air person will ever again. In 1979, he told NBC he was quitting and the network freaked out. In the end, Carson wielded the power, regaining ownership of the show, keeping all of his vacation (8 or 10 weeks, I believe), only four new shows a week plus trimming it from 90 minutes to 60. Carson's deal was like the Treaty of Versailles.
Even as NBC was clearing $50 million a year in profit off Carson's show, the network wanted to make sure whoever succeeded Carson would be a "yes, man" and agreeable to their ideas.
Enter Letterman. Clearly the more talented and edgy host compared with Jay Leno in the 1980's. Like Carson, Letterman was an angry perfectionist about his show. He was miserable to be around after each episode of Late Night as he would watch the tape and dig and pry for any mistakes that anybody did.
In television, that's how the elite performers operate. They're ashholes and pricks. Doesn't make it right but they hold themselves to an incredibly high standard.
NBC pashed on Letterman because he was a prick and Leno would be more than happy to fly to Toledo, Topeka and Tulsa to shake hands with the news anchors and do promos. (I met him twice at different NBC stations in my career -- good guy and truly loves what he does).
Once Letterman landed the CBS job in 1993, he tried hard for a month and then flipped it into cruise control, the only exceptions being his heart surgery and post 9/11.
NBC is determined NEVER to let another prick sit in the Tonight Show seat, never to have another host with the juice to own his own show (like Carson or Letterman at CBS), one who "knows his place". Letterman made a living out of ripping NBC and GE from about 1985 on and that actually sealed his fate to never host the Tonight Show.
Conan O'Brien was out of the Letterman mold, never afraid to rip the network and also not shy about ignoring input from network executives. Once the ratings crapped out, Conan was done.
Fallon will do whatever it takes to hold onto the job, similar to Leno. He won't be edgy but he'll be agreeable to new ideas -- or at least he will flatter the executives.
NBC also benefits from having a "D-League" of comedic league on SNL. The second they don't like Jimmy Fallon, they can just look to whomever is doing Weekend Update as a possible replacement.
