I was working on a code using libmosquitto v3.1.x and I couldn't find a method to capture the dup flag of a message.
Nb1: From my understanding dup flag indicates that the message was already send but was not acknowledged by recipient.
Nb2: I was referring https://mosquitto.org/api/files/mosquitto-h.html for libmosquitto API descriptions.
At time of writing of this answer dup flag functionality is not supported by libmosquitto (v1.04.15).
Its better to use an advanced library like Paho if you are working ony anything serious.