pcap_set_rfmon succeeds but doesn't actually work

I'm currently trying to set up a simple packet sniffer with libpcap on Ubuntu 20.04.3 LTS and facing a lot of confusion over setting monitor mode with pcap_set_rfmon(). A trimmed version of my code and the compilation command I used is below:

g++ trimsniff.cc -g -o tsniff -L/usr/local/lib -lpcap

Code:

#include <iostream>
#include <pcap/pcap.h>
#include <string>
#include <cstdlib>
#include <cstring>


using namespace std;

int main(int argc, char *argv[])
{
    //Declare needed variables
    const int MAX_NAME_LEN = 20;
    char errbuf[PCAP_ERRBUF_SIZE];
    char dev[MAX_NAME_LEN];
    pcap_if_t *alldevs;
    pcap_if_t *alldevsp;
    pcap_t * handle;

    //Check Libpcap version number
    cout << pcap_lib_version() << endl << endl;


    //Initialize the library for local charactr encoding & error check
    if(pcap_init(PCAP_CHAR_ENC_LOCAL, errbuf))
    {
        fprintf(stderr, "Couldn't Initialize pcap; %s\n", errbuf);
    }
    else
    {
        cout << "PCAP Successfully Initialized" << endl << endl;
    }
    
    //trimmed version of device selection code, this assumes an 
    //available device was specified in the command line call 
    //(I make sure of this in the full code without error) 
    strcpy(dev, argv[1]);

    cout << endl << "Selected Device: " << dev << endl << endl;
  
    //Open device for sniffing
    handle = pcap_create(dev, errbuf);

    //Try setting monitor mode and error check, trimmed down to the error I'm facing 
    int mm_set = pcap_can_set_rfmon(handle);
    if(mm_set==0)
    {
        fprintf(stderr, "Error setting monitor mode: Device doesn't have MM capability\n");
    }
    else
    {
        if(!pcap_set_rfmon(handle,1))
        {
            cout << "Monitor Mode Enabled, pcap_set_rfmon(...) == 0" << endl;
        }
    }

    cout << endl;
    
    //Using pcap_set_rfmon() here to illustrate issue, this will output a 0 
    //indicating success but the pcap_activate() error check contradicts this
    cout << pcap_set_rfmon(handle,1) << endl;

    //Activate the interface for sniffing
    if(pcap_activate(handle))
    {
        cout << endl;
        pcap_perror(handle,"Error");
        cout << endl;
        pcap_set_rfmon(handle,0);
        pcap_activate(handle);
    }


    pcap_close(handle);

    return 0;
    
}

My device is certainly capable of monitor mode as I've used terminal commands and aircrack-ng to monitor unassociated network traffic successfully in the past.

But when I try to use the libpcap functions, pcap_set_rfmon() will return 0 as if it succeeds, while pcap_can_set_rfmon() contradicts this and returns 0 indicating that monitor mode cannot be set. The output of my trimmed code is below, the line with a zero is the output of pcap_set_rfmon(), indicating success.

libpcap version 1.11.0-PRE-GIT (with TPACKET_V3)

PCAP Successfully Initialized


Selected Device: wlx00c0caadea0a

Error setting monitor mode: Device doesn't have MM capability

0

Error: That device doesn't support monitor mode

The last error message comes from calling pcap_activate() ( using the libpcap error printing function pcap_perror() ), after trying to set monitor mode with pcap_set_rfmon().

Does anyone know where this contradiction comes from and/or how to resolve it?

After looking around a bit, this is apparently a problem with Linux based systems. Libpcap needs to link with libnl to properly set monitor mode with pcap_set_rfmon(), and this doesn't happen, likely due to conflicting versions of the libnl library. This function works fine on my Mac for setting monitor mode, but in Ubuntu I have to use the system() function with ip link and iw console commands as a workaround. So long as you do some OS detection beforehand it's trivial to have your program decide which method to use.