djangodjango-rest-frameworkpaginationdjango-pagination

Django Rest Framework Cursos pagination with multiple ordering fields and filters


I have an issue with DRF, CursorPagination and Filters.

I have an endpoint. When I access the initial page of the enpoint I get a next URL

"next": "http://my-url/api/my-endpoint/?cursor=bz0yMDA%3D&date__gte=2025-04-25T10%3A00%3A00Z&date__lte=2025-04-26T10%3A00%3A00Z"

When I access this URL I get a previous URL

"next": "http://my-url/api/my-endpoint/?cursor=bz00MDA%3D&date__gte=2025-04-25T10%3A00%3A00Z&date__lte=2025-04-26T10%3A00%3A00Z",
"previous": "http://my-url/api/my-endpoint/?cursor=cj0xJnA9MjAyNS0wNC0yNSsxMCUzQTAwJTNBMDAlMkIwMCUzQTAw&date__gte=2025-04-25T10%3A00%3A00Z&date__lte=2025-04-26T10%3A00%3A00Z",

Now when I try to access the previous URL, I get an empty result list.

Here is the code for the endpoint

class RevenuePagination(CursorPagination):
    page_size = 200
    ordering = ['date', 'custom_channel_name', 'country_name', 'platform_type_code', 'id']


class RevenueFilter(django_filters.FilterSet):
    class Meta:
        model = Revenue
        fields = {
            'date': ['lte', 'gte'],
            'custom_channel_name': ['exact'],
            'country_name': ['exact'],
            'platform_type_code': ['exact'],
        }


class RevenueViewSet(viewsets.ModelViewSet):
    permission_classes = [HasAPIKey]
    queryset = Revenue.objects.all()
    serializer_class = RevenueSerializer
    filterset_class = RevenueFilter
    pagination_class = RevenuePagination

    @revenue_list_schema()
    def list(self, request, *args, **kwargs):
        return super().list(request, *args, **kwargs)

From what I get, the problem seems to be that the previous URL cursor (cj0xJnA9MjAyNS0wNC0yNSsxMCUzQTAwJTNBMDAlMkIwMCUzQTAw) gets decoded to r=1&p=2025-04-25+10:00:00+00:00, which is not right, because only the date field is in the cursor, while I have specidied 6 fields in the ordering of the pagination.

I tried it narrowing down the order to ['date', 'id'], but it does not work.

All the fields in the ordering are actual DB fields (DateTime, Char or Integer), no properties.

I am really struggling, tried debugging with Google AI and ChatGPT, both came to a dead end.

EDIT: I went in the DRF source code, and I found that the CursorPagination._get_position_from_instance only takes into consideration the first field in the ordering list.

def _get_position_from_instance(self, instance, ordering):
    field_name = ordering[0].lstrip('-')
    if isinstance(instance, dict):
        attr = instance[field_name]
    else:
        attr = getattr(instance, field_name)
    return str(attr)

This is really strange, as per documentation the CursorPagination.ordering supports multiple fields. Or am I wrong?

EDIT2: I narrowed the problem down to having date value repeating, and filtering on the date value.

The filter is date__gte=2025-04-25 10:00:00, and the first 10 pages have the same date value '2025-04-10 10:00:00'. The CursorPagination seems to be taking the first date value from the second page, and using that in the previous cursor. I think it needs something more unique. The combination fields provide uniqueness, but it does not work.


Solution

  • ordering can (and often should!) be multiple fields.

    But you're right that only the first field is stored in the cursor token. However, the other fields are still applied:

    # rest_framework/pagination.py
    
    class CursorPagination(BasePagination):
        [...]
    
        def paginate_queryset(self, queryset, request, view=None):
            [...]
    
            '''
            The below part is where the ordering fields come into play.
            They are not stored in the token, instead they are reapplied 
              on every hit of the paginator.
            '''
            # Cursor pagination always enforces an ordering.
            if reverse:
                queryset = queryset.order_by(*_reverse_ordering(self.ordering))
            else:
                queryset = queryset.order_by(*self.ordering)
    

    So this part of the investigation is a red herring.

    The lacking uniqueness across a large number of rows in the primary ordering fields is the primary part of this issue - if we exclude design decisions made by the DRF team.

    You were on to the right idea at the end of your post. The cursor reversing logic is roughly as follows:

    1. Take first element of current page
    2. Generate a condition on the primary ordering field that excludes #1
    3. Generate page with #2
    

    For ordering fields that are unique (or are unique "enough" that multiple pages won't have the same ordering value), this is perfectly fine.

    But for your dataset, the condition that rules out elements from the current page also rules out all the elements that are supposed to be on the previous page.

    Initial page:

    initial page

    Next page:

    next page

    Previous page:

    previous page

    SQL:

    enter image description here

    WHERE "event_event"."date" < '2025-04-25 08:00:00'
    

    This is the condition that's built to rule out the 2nd page elements, but since the 1st page elements have that same value they get ruled out too.

    The reason the subsequent ordering fields "don't matter" in this specific instance, is because that ordering doesn't take place until after the SQL has been generated, and since the query returns 0 rows, there's naturally nothing to order.

    With your dataset, this isn't easily fixable with cursor pagination. You'd have to build cursor logic where the token is encoded with multiple fields instead of just the one, and that has its own complications. I don't know what all of them are, but I'm guessing there's either a limitation in the ORM that makes certain condition-combinations tricky to write, and/or an issue with how to generate/map multi-field orderings into proper SQL in a universally applicable way that can also be reliably reversed.

    If you really want to hack it with cursor pagination, you could store the "current page" URL on the client's localstorage and use that URL for previous instead of letting DRF reverse the token by itself.

    But if you can't switch to a more unique primary ordering field, my best recommendation is switching to page number pagination.