First of all, there is no realistic way you can truly stop anyone from saying what they are going to say. People who are filled with hate and verbal bile will sooner or later be found out and their personality all but seals their social fate.
If it were me, I would not worry about that which I have little or no control over. I would continue to live an honorable life and conduct myself is such a manor that would make my children proud of me. After all, the things which we DO in this life greatly outweigh the things which are being said.
You want your kids to know and remember you as a kind a loving man who has the wisdom to keep negativism away from your children and to impart only kindness and consideration for everyone and everything.
And soon your children will say, “I love dad, no matter what you say”