Assign User-objects to a Group while editing Group-object in Django admin

In the default Django admin view for user-object (edit user) one can edit the user's group memberships. What if I wanted this the other way around also? I.e. in the group editing page one could select the users that belong to the group being edited.

As I see this, Django doesn't have a ManyToMany mapping from Group to User object which makes it impossible(?) to implement a ModelAdmin class for this particular case. If I could make an additional UsersOfGroup model class and use it in the Django's Group model's ManyToMany field as a through-attribute, there could be a way.

Any ideas, is this possible to implement using ModelAdmin tricks or do I just have to make a custom view for editing groups?

I have checked these two other questions, but they don't quite do the same thing:

Assigning a group while adding user in admin

and

Show group membership in admin

Updated: The answer from Chris was almost there. :) The group has a reference to the users set, but it's called user_set, not users. So these are the changes I made:

if self.instance and self.instance.pk:
    self.fields['users'].initial = self.instance.user_set.all()

and

if group.pk:
    group.user_set = self.cleaned_data['users']

yourapp/admin.py

from django import forms
from django.contrib import admin
from django.utils.translation import ugettext_lazy as _
from django.contrib.admin.widgets import FilteredSelectMultiple

from django.contrib.auth.models import User, Group

class GroupAdminForm(forms.ModelForm):
    users = forms.ModelMultipleChoiceField(
        queryset=User.objects.all(), 
        required=False,
        widget=FilteredSelectMultiple(
            verbose_name=_('Users'),
            is_stacked=False
        )
    )

    class Meta:
        model = Group

    def __init__(self, *args, **kwargs):
        super(GroupAdminForm, self).__init__(*args, **kwargs)

        if self.instance and self.instance.pk:
            self.fields['users'].initial = self.instance.users.all()

    def save(self, commit=True):
        group = super(GroupAdminForm, self).save(commit=False)

        if commit:
            group.save()

        if group.pk:
            group.users = self.cleaned_data['users']
            self.save_m2m()

        return group

class GroupAdmin(admin.ModelAdmin):
    form = GroupAdminForm

admin.site.unregister(Group)
admin.site.register(Group, GroupAdmin)