How do I tell him I don't want to live next door to his family?

Although we haven't seriously discussed future plans, this is still something that may cause problems down the line. My boyfriend recently built a house for himself, his parents next door and a cottage for his grandparents behind that. They all come and go as they please and although I love his family, I would be miserable not having privacy or alone time with him. He is perfectly content spending the rest of his life there, and even tells me that his grandparents cottage is his "retirement plan". I've told him I don't see this being my "forever house" for various reasons, including size, personal taste and the fact that I want to build a life WITH him, not find room to fit into his, but have been to afraid to admit that the biggest reason is because of the close proximity of his family. I don't want to hurt or upset him but I'm not sure how to honestly tell him how I feel...Any gentle suggestions?