Does this mean I have to abandon my family/friends?

Sometimes our biggest lessons come to us in the form of our nearest and dearest. We wouldn't recommend abandoning your family. However you certainly have power how much of time you spend with them. If you notice you're drained or have negative emotions when with them, then limit your time.

On the other hand, it's good to notice as to why they're bringing up such negativity in you. What emotions do you feel exactly? Instead of them dragging you down how about you start working on yourself and becoming the best version of yourself that whenever you guys hang out you automatically start lifting them up because of how bright you shine.