XOR toggle an Enum value in or out of an EnumSet

I am refactoring some code that used low-level bit manipulation and masking to reimplement it with an EnumSet. The incentive is: having a much more maintainable and readable code. I find it appropriate to abstract the logic away from low-level implementation like bits and memory units, while trading some minor efficiency loss (if any).

I already wrote the base methods to set, unset various states to replace bits. But I also need a method to flip a flag (set if unset, unset if set).

Here is a simplified subset of my class. It handles the state of input restriction per side of a block in a MinecraftForge mod (if context matters). Most of the time, player interacts with the setting of the side restriction, to flip it one way or the other (thus using the flip() method).

public class InputRestrictor {

    protected EnumSet<ForgeDirection> sideRestricted = EnumSet.noneOf(ForgeDirection.class);

    public void set(final ForgeDirection side, final boolean isInputOnly) {
        if (isInputOnly) sideRestricted.add(side);
        else sideRestricted.remove(side);
    }

    public boolean get(final ForgeDirection side) {
        return sideRestricted.contains(side);
    }

    public void flip(final ForgeDirection side) {
        set(side, !get(side));
    }
}

I think if the EnumSet had a native method to flip an enum value in-out of the set, it would be more efficient than first fetching the boolean state, inverting it and setting it or removing it conditionally.

I read in this post Why an EnumSet or an EnumMap is likely to be more performant than their hashed counterparts?, an EnumSet is internally implemented using a bit-mask or bit-field. Would be convenient if one could just toggle XOR one or more enum values (in or out of the set) in one step like you'd do with:

byte bits = 0b000111;
bits ^= (byte) 1 << 2; // inverted bit 2: bits = 0b00011
bits ^= (byte) 0b111111; // inverted all at once: bits = 0b111100

Are there more direct toggling methods beside going low level with bit manipulation?

No. EnumSet only exposes the basic Set methods, which do not include a toggle of the form you describe.

I would suspect that the reason it does not provide such a method is that it would be very rarely used. Most usages of Sets -- certainly those that'd require high performance -- seem likely to only need to add and remove.