Fortress is a great defense against individual enemies--the Knights, Swindler, Saboteur, Rogue--but not against massed enemies--Goons, Militia, Mercenary, Followers. You'd think that the reverse would be true, where a lone spy could sneak into a fortress but a large army could not.
(Yeah, I know the edge cases: it actually makes sense that a Fortress can't stop magic attacks (Witch etc), and it also can't protect against Cutpurse--but throw me a bone here).