Give The Recognitions another chance. It takes a couple dozen pages to suss out, but is to postwar American literature what Pollock was to postwar American painting. Of course, you might more naturally favor Gaddis' JR.
The New York Times reviewed a book about Alexandre Dumas' grandfather, who was the inspiration for the Count of Monte Cristo, on Sunday. It looks awesome. My four that I could never choose between are Infinite Jest, For Whom the Bell Tolls, Gravity's Rainbow and Tender is the Night.
I think the idea of a bracket like this is absurd. The top spot for a person should be a multiway tie among the transcendent reads the person has had. The Telltale Heart by Dickens, The Mysteries of Pittsburgh by Chabon and Anywhere but Here by Mona Simpson were transcendent reading experiences for me. Different in many ways, but tied in the most important criterion.
Armchair, I would go with Semi-Tough. Remember reading it in 1974 or 1975 and was mightily impressed.