No. Just can't seem to get my villagers around the red masks. Will keep trying.
Definitely keep trying. You'll have to be sneaky about it to get around those Red Masked Heathens. Yes, they stand in front of the statue and will not allow anyone near, but if you watch them there are times when they are both on one side of the statue.
When this happens, sneak your villagers in there (trying to make sure they don't see the Red Masks) and get them to start dismantling the totem. Put all of your adults on it.
I believe they will take cover when it's raining so that's also an opportunity to dismantle the totem.
As others have said, lightening also works, just make sure your villagers are not too close or they will get zapped too.
Good luck! You can do this!