Before he betrayed the team, Booker would sometimes join Joe and Nicky in bed. Nicky likes being dp'ed and Booker was the obvious choice to join them. He wasn't part of their relationship, this was purely sex. They are not in love with him, but do love him like a brother.

After Booker gets brought back into the fold early, he expects everything to remain the same. The first time Booker reaches out for Nicky, Joe grabs his wrist and pulls him away.

They tell him there is no way they are about to trust him in their bed any more. What he did means he will never be welcome to lie with them again.

This doesn't end with them forgiving him, but him realising exactly what he screwed up and having to live with it.