Darwin, power, and discrimination

 Suppose you lived in a society where certain people were treated as inferiors. The justification this treatment is "It's a dog eat dog world, and as Darwin said, it's survival of the fittest. We have the upper hand, and we have to power to make these other people do what we want. Who's to say that there's anything wrong with that?"

The Declaration of Independence says that we were endowed by our creator with certain inalienable rights. If you replace the word "creator" with "evolution" in that statement from the declaration, you end up with nonsense. We clearly didn't evolve equally. So, how would you argue against discrimination from a Darwinian perspective?