ListView: hide Contextual Action Bar

I'm following the official guide for using contextual action mode like this:

    listView.setChoiceMode(AbsListView.CHOICE_MODE_MULTIPLE_MODAL);
    listView.setMultiChoiceModeListener(new AbsListView.MultiChoiceModeListener() {

        @Override
        public void onItemCheckedStateChanged(ActionMode mode, int position, long id, boolean checked) {
        }

        @Override
        public boolean onCreateActionMode(ActionMode mode, Menu menu) {
            MenuInflater inflater = mode.getMenuInflater();
            inflater.inflate(R.menu.shelf_context, menu);
            return true;
        }

        @Override
        public boolean onPrepareActionMode(ActionMode mode, Menu menu) {
            return true;
        }

        @Override
        public boolean onActionItemClicked(ActionMode mode, MenuItem item) {
            // some processing...
            return true;
        }

        @Override
        public void onDestroyActionMode(ActionMode mode) {
        }
    });

My listView is inside a Fragment which is inside a ViewPager, so I want to hide the Contextual Action Bar when the fragment becomes invisible. But how do I do that? I call listView.clearChoices() to clear the selection, but the CAB is still visible:

@Override
public void setUserVisibleHint(boolean isVisibleToUser) {
    super.setUserVisibleHint(isVisibleToUser);
    if (!isVisibleToUser) {
        listView.clearChoices(); // CAB is still visible.
    }
}

After some research I found a hacky way to do that:

@Override
public void setUserVisibleHint(boolean isVisibleToUser) {
    super.setUserVisibleHint(isVisibleToUser);
    if (!isVisibleToUser) {
        listView.clearChoices();
        listView.setChoiceMode(AbsListView.CHOICE_MODE_MULTIPLE_MODAL); // this strange hack dismisses the CAB.
    }
}