If you percieve your girlfriend cheated on you, because of some viscious rumor, so you go and kill her, it did harm.
If some stupid goth kid thinks everyone hates him because he has social anxiety and slits his wrist, it did harm.
If some schitzophrenic person thinks his mother is part of the FBI who are possessed by aliens so they try to save her by injecting random drugs into her and she dies, it did harm.
Just because, to you, their perception may not be real, does not mean it doesn't effect this reality in a very real way, and as effecting this reality it will effect yours;
I.e., the schitzophrenic person who killed his mother also killed his best friend, because they were in cohoots, this best friend of his was your younger brother. Very real effect on your reality, something you can understand.
In a simpler, more chaotic and complex way, somewhere along the way, some crazy unrealistic person did something to horribly effect someone, who horribly effected a 100 other people, who horribly effected another 100 people, and so on, until that single chain of events ends up hitting you, and effects your reality. So no matter how you look at it, you are always just going to be a result of people who don't fit into your perception.
That is also a way to explain karma
I'm sure you can agree, that someone who believes something untrue, and false, is deluded-- whether or not they are insane-- and despite this fact, people will act on their delusions... so even if there hav ebeen no crazy people in some way contacted with you, there is going to be someone who believed something false, or merely functioned off of their perception.
In effect, there is nobody on this earth who doesn't fun ction off of perception and fallacy.
Like a christian who raises a child, or a buddhist. Or a mormon. or a jew. One of those has to be "Wrong" by most standards of reality.