How to prevent retrieval of all entries from bidirectional ManyToMany collection while adding new entry

I would like to know how to prevent retrieval of all entries from bidirectional ManyToMany collection while adding new entry into that collection. I have User entity, that has bidirectional relationship with UserGroup entity. When I add group in User entity it add User to group as well as the UserGroup is parent. The problem is that if I add a lot of groups (let say 20) to user, then user is being added to each group as well and all users (100+ per group) are retrieved from each group. It makes transaction to take a lot of time and shows significant lag while I only want to insert UserGroup into User's userGroups collection and have no need of selecting other users from those groups.

User.java

@ManyToMany(mappedBy = "users")
private Set<UserGroup> userGroups;

public void addGroup(UserGroup userGroup){
this.userGroups.add(userGroup);
userGroup.addUser(this);
}

UserGroup.java

@ManyToMany
@JoinTable(name = "group_users", joinColumns = @JoinColumn(name = "group_id"),
        inverseJoinColumns = @JoinColumn(name = "user_id"))
private Set<User> users;

public void addUser(User user) {
   this.users.add(user);
}

Method used

@Transactional
public void addGroupsToUser(UserAddGroupsCommand userAddGroupsCommand, String username) {
    User user = userRepository.findByUsername(username);
    if (user != null) {
        List<UserGroup> userGroupList = userGroupRepository.findAllById(userAddGroupsCommand.getId());
        for (UserGroup userGroup : userGroupList) {
            user.addGroup(userGroup);
        }
        userRepository.save(user);
    }
}

So I can see in console, that all join table entries are selected for the group I am iterating at

EDITED: this is the code that execuded and I am not sure why.

select users0_.group_id as group_id1_0_0_, users0_.user_id as user_id2_0_0_, user1_.id as id1_7_1_ 
from group_users users0_ 
inner join User user1_ on users0_.user_id=user1_.id 
where users0_.group_id=?

Try custom query to gain better performance

from UserGroup g left join fetch g.users where g.id in :IDS

This way users in each group will be loaded in one query and not in N+1 queries.

Changing @ManyToMany(FetchType.EAGER) is not efficient, because users will be always loaded with UserGroup, also when not used.

Using @EntityGraph is same as join fetch query.