I think this behaviour is more obvious when you have more villagers, and especially when you have a lot of villagers set to breeding