I'm torn.on this. Not every good team is going to play amazing every single game. It's just common sense.
I think if a team has a difficult game don't penalize them. Now if they have one, then maybe another, not necessarily back to back, then that could be a pattern the voters look at.