Separate the baby boys out at 4 weeks. The girls can continue to live with mom.
Although most girls are not sexually mature at 4 weeks, it's just not worth it to risk it (and I've heard exceptions to this "rule"). By leaving the girls with mom, and the boys together- no one is left lonely!

Just keep an eye on the boys for any squabbling since often boys will not get along after sexual maturity. Many boys who live together from pinky-hood do, however, so it's nice to give them a chance at having friends.
