Are there any awkward moments where you're walking up/down your driveway, or getting out of your car while he's out there as well? @PB You know.. random shit encounters within shouting distance.
You can't just go and punch your neighbor. 
I've lived here 20 years and have seen him maybe 3 times. I'm not certain I could pick him out of a police lineup.
If I were to run into him, and ascertain his identify, the conversation would go something like this: ''Stay the fuck out of my yard, don't reach into my yard,
cut any branches off at the property line - don't just reach as far as you can and snap them off, and don't throw your fuckin' trash into my yard. Make sense?'' Probably just about exactly like that.
I once helped his 100 year old mother when she came over and told me she locked herself out - he was on vakay in Mexico, good thing I was around. I helped him cut down a tree once, when I saw him and some friend (the brother in law?) out there - to make sure it didn't fall on my roof.
He's supposed to be a retired engineering professor from UC Berkeley, but seeing that tree project and some of the other things he's done, he can't possibly be an engineer.
His mother owns the house - she's Japanese and lived most of her life in Hawaii. From the one time I saw and talked to her I think she regrets selling her beautiful home, furniture, etc, and moving here. He's got a sister that lives around here somewhere that visits once in awhile, I wouldn't recognize her either. They're very nondescript.
He's got the same model red Ferrari in his garage as the Magnum PI car, never seen him drive it - I saw it when the housekeeper (possibly the sister?) locked herself out (fortunately the garage door was open, getting into the house from there was a snap) and once or twice since when the garage door was open.
That's about it.
But they keep quiet, so that's 99% of being a great neighbor.
