So far I've noticed that those who dislike flowers won't enjoy the garden. However, if you take them there in your hand, the status bar would say that they feel very well there, just as every other villager. Also the ones who don't like to swim would try to leave the water as soon as possible.