I like Cameron and lyman's ideas, but why not go one step further:
forget about time completely. You are immediately moved to the inactive
list (which is still shown) if you decline a challenge while not having
any other game of that type going.
Hey, that *was* my idea :)
I think this should do the trick. One drawback is that players may be
seen as inactive if they refuse a challenge in a particular game even
if they have say 50 games of other types currently underway (ie a
legitimate reason to decline).
But this is no real drama. The declining player might receive a
message along the lines of: "You have been place in the Inactive list
for [game]. Start a new game to be reinstated as an Active player."