A lot of great suggestions here, a few items I would mention.
1. Don't think short term on a deer property - think long term
2. Make a list of wants (ponds, fields for planting plots, AG fields near or far, areas for orchards, hard mast,etc).
3. Don't think that other adjacent land won't ever be for sale - be ready for it to sell and move quick!
4. Consider as many factors as you can think and make a pros/cons list - that are not deer related
5. Set a list of goals - do you want to see a LOT of deer, GREAT bucks? A mix of both?
My family bought 60 acres 10 years ago (give or take). We started planting plots, fruit trees, meeting some neighbors, etc. We do have Amish in the area but none are touching our farm. We made our hunting as good as possible and killed a couple decent bucks over the years. We always wanted to see more does so we have limited harvest on the does. At times I wished I had a farm in an area with a higher deer density but at the same time I have fallen in love with working on my slice of heaven. We were fortunate enough to purchase the adjacent 170+ acres this past summer, it is amazing how many more deer we have seen this year now that we are in control of a large chunk of ground.
My point is really think it over, don't let the "big" buck trail cameras on fancy real-estate sites make get over excited. Remember when you buy a piece of ground you are now invested for awhile, so mine as well put as much thought into as possible. If you think you want 50 acres, try to buy 75, if you want 75 try to buy a 100. If you are like me, you'll always want more.