I finally beat them with first lightning and then got my peeps on the rocks with a master on the totem. When the red guys started coming back, I hit them with bees until my points ran out. The yellow heathens I fended off with a decoy. When the blue heathens come back to rebuild, I put a peep on them and the blue heathens would flee. Then did it all three or four more times when I had enough points to try again.

The second time I played I noticed that totem had NO red heathens guarding it so it was the first thing I went after. After the first conversion attempt on the blue heathens, any subsequent attempt sends them fleeing so it was easy to drop peeps on them to get them away. I then put all my peeps on removing the rubble and dropped a peep on any returning blue heathens. I skipped the totem so did not need a master builder.