The biggest part that gets me about that way of thinking is that you actually believe that shit. A fuggen tree is not bait, it's a fuggen tree!
No different than a feeder. Both distribute a food item that attracts an animal. The animal is being lured to that spot by........ You guessed it. Bait.
Now with an apple tree you might could argue that you aren't baiting but rather the tree is. Which is a silly argument if you're hunting over it. Hunting over apples that fell off a tree, or hunting over apples that fell off a tree and you moved, is no different. Both will bait an animal to a spot.
That's like trying to say that hunting over dumped corn is baiting. But hunting over corn that fell from a feeder isn't baiting. The end result is corn is on the ground attracting animals. Just as apples are on the ground attracting animals. How it got there is irrelevant.
The reason the law draws the line where they do is enforceability. You can't fine and apple tree. They define it that way not because it isn't baiting, but rather that pesky burden of proof.