No matter what kind of positive impact you have on the world, there will always be someone who thinks you're a selfish jerk. And maybe you are!
So what?
Go make a dent in the universe anyway. Take up your hammer and start making some noise. Drown them out with the wonders you'll create.