How to deal with dating interracially when ur family disapproves?

so the love of my life is black, i am white. we dated for a couple years a couple years ago. and always stayed friends. we are soul mates. but the problem is that in the meantime we were broken up i found someone else and got pregnant and had a baby with him. he is not my type at all and ive stayed with him two years for our daughter. well now the other guy is sorta back in my life, today i went and saw him and lets just say that some old flames got rekindled. well i want to leave my babys daddy but im scared that if he finds out im with the other guy he will freak b/c he hates black ppl and this man USED to be a big drug dealer, (now he has a job and is settled down alot). anyways, my dad is the same way. he will flip if he found out i was back with him and his granddaughter was around him. my dad never liked us together before but now would really not like it b/c of his granddaughter being around him. but should i really care or go with what makes me happy? my daughter would be in a total drug free environment, no worrys there, the only problem would be my baby daddy and my own dad would hate that i would have her around him. can anyone help?