I agree with your boyfriend, it does make for an awkward situation. And this will most likely be an awkward situation for any boyfriend. An it will be an awkward situation for your ex's new girlfriend too.
Think about how you would feel if you ex or ex's started showing up with gf's at your family dinners, vacations, etc.
I don't doubt this particular family is super nice, etc., but at some point it may be best to not be so totally involved with them.
I think you should enjoy your current boyfriend and his family/friends. And also, spend time with yours.