It is because I see trespassing in a slightly different light. I'm a little uncomfortable placing it in the same category as aggression and theft.
First of all, as I implied in my definition of "trespasser", if you are there with the owner's permission I don't consider you a trespasser. Not even if you failed to abide by the rules that gave you access and if the permission was contingent on those rules being followed. It may not be nice of you, and you should immediately leave when asked to do so. Regardless of whether the rules are legitimate or not, you broke an agreement, but you are not trespassing.
This is the reason I think personal property rights trump real estate rights, a la "Bubble Theory".
The bigger point and the core truth is that you can trespass without causing any harm (offending someone is not "harm") and without having any ill intent.
A lost person stumbling around in the woods can easily become a trespasser but may not damage the violated property in any way and may not have intended to trespass to begin with.
I have also known of trespassers who knew they were entering private property but who had no ill intent or ulterior motive at all, and who even assumed they would be welcome.
I have been on both sides of the equation. I have dealt with trespassers of both types on many occasions and never felt the need to get tough with them. No harm, no ill intent- no real problem.
This is why, unless there is more to the story, I don't view trespassing as a wrong on the same level as aggression and theft. But... maybe I'm wrong to think that way. At least it's something to mull over.