If you look you'll see a color is missing. Have two villagers bring the two colors of dye that make the missing color to the pool at the same. when you make that color the heathen(s) will leave the pool giving you some time to dismantle it. You'll have to do it several times depending on how many villagers you have working on it.

before I had the clothing hut, using bees. Where do I find the dyes now? I don't see them anywhere.

the dye pots will never appear. They are used only to provide, as Arthur put it, "an elegant solution" to this puzzle.

So no clothes dyeing? Awww...

"Fog of Doom" power. The I then placed 20 of my villagers on it and they began to dismantle it. They couldn't see the scary heathens. They were done before the fog disappeared