I get them too! One time a teacher said something and it reminded me of something traumatic that happened a few months before then. It was instantaneous, it was like I was back there again- I could see myself in that room again, see the people I was with in that situation and even feel those same feelings all over again. I walked out of the class crying, I wasn't necessarily anxious, but more shaken and upset.
I've never actually thought about calling them 'ghosts' before, that's an interesting way to look at it. I usually call them flashbacks or 'unpleasant memories' haha.
But the main thing with your situation is you don't have to worry about that colleague anymore, they're no longer a part of your life- just as it should be!
Kaitlyn