So everyone who's played in well established servers that haven't merged understand that late game, players leave and new players don't start because of player stronghold average lvl 39/40.
So many cities are left empty with only guard towers as people move around or join forces to get imperium or control of other towns.
So my suggestion is that if there are no player strongholds in a city that the guard towers slowly loose troops ( kind of like they Regen). So they would all stop regening troops and wall HP as long as no players were there. And after x amount of time the Garrison's would slowly leave the guard towers.
Obviously it wouldn't be extremely fast decline so wars can't abuse the mechanic. But the relative time of all wars shouldn't compound enough to make a difference in an active fight. Only cities that are left relatively abandoned.