Another thing that happened right around the time the ratings started dropping so fast, and this is probably minor but pop culture can be a cruel mistress --- Talladega Nights came out and was a hit movie and, for mass pop culture, Ricky Bobby became the face of Nascar.
And, sure, it was laughing with Nascar and Nascar fans and drivers ... but it was mostly laughing at them. And most of the jokes were about how backwards, poor, stupid, and low class Nascar and Nascar fans were. Nascar was the butt of most of the jokes, not just the setting for them.
I thought it was a funny movie. But I could see how it could cause some casual fans to not want to be associated with something that everybody was laughing at as a stupid thing for stupid people. This is probably a bigger factor for young fans just being exposed to the sport and more swayed by pop culture.
It's impressive how Talladega Nights managed to come out right around the peak of Nascar's popularity.