I was on my way to the Dwarven Ruins and I was tired of fighting every guard I came across, so instead I thought "hey I'm faster than they are when they start attacking" and just ran past everyone all the way to the top of the mountain. I had some potions I wanted to save for the dungeon too.
IT TURNS OUT, guards not only don't forget that they saw you, but they also never de-spawn... and their pathfinding is excellent somehow. So much for the potions I had stockpiled.
Spoiler:
