Is it ok for.....(fraternity)?

If they were worthy people to be around, they wouldn't care. And please don't go changing yourself or deny who you are because you're a person of color. Be proud of who you are and have confidence - and other people will feel confident about who you are too and they won't try to take advantage.