C++ polymorphism across boundaries in shared libraries

I am developing a simple logic-gate simulation. For each logic gate type the following class is implemented:

class Gate {
public:
    explicit Gate();

    virtual ~Gate() = default;

    virtual void update() = 0;
};

That means each gate type for example an AND gate can implement the Gate class and implement the logic for this gate inside the update method. These gate objects are then connected by pins and form a graph. When updates in a gate occur, all child gates are updated recursively. So far so good.

My goal now is to create a simple plugin system using shared libraries, which allows others to write own gates for this simple simulation. I now have issues regarding ABI compatibility. To solve all ABI issues I decided to use only C89 as the interface between plugin and main application. However, how should this work with my gate implementation? My plugin would have to implement the Gate class and implement the update method, which is okay, as the virtual Gate class is inside a header file both the plugin and the main application have access to. But I cannot share instances of the custom Gate implementation from the plugin with the main application, as it is a C++ object and it would break the ABI.

In the end it is a general issue. What if I want plugins to be able to implement classes to define custom behavior while keeping the interface between plugin and main application ABI compatible

How should I approach this?

If you want great ABI compatibility you have to stick to C at DLL boundary. I think the hourglass pattern might be a good fit here. Implement C-based manual OOP objects for the boundary, and wrap it up on both ends into something more palatable. Something like:

#include <iostream>

// ABI boundary interface SDK
extern "C" {
    struct PLUGIN_Gate;
    struct PLUGIN_GateVtbl {
        // add cdecl CC maybe
        void (*Destroy)(PLUGIN_Gate*);
        void (*Update)(PLUGIN_Gate*);
    };
    struct PLUGIN_Gate {
        const PLUGIN_GateVtbl* vtbl;
    };
}

This is basically the same interface but in plain C, and with C functions. Depending on a lifetime management style, you might want Destroy or not; just make sure allocation and deletion always happens on the same side of ABI boundary.

// plugin side C++ interface, declared in SDK header
class IGate : PLUGIN_Gate {
public:
    explicit IGate();
    virtual ~IGate() = default;
    virtual void update() = 0;

    PLUGIN_Gate* handle() {
        return this;
    }

private:
    static void Destroy(PLUGIN_Gate* Self) {
        delete static_cast<IGate*>(Self);
    }
    static void Update(PLUGIN_Gate* Self) {
        static_cast<IGate*>(Self)->update();
    }
    static const PLUGIN_GateVtbl IGate_vtbl;
};

// In SDK C++ source
const PLUGIN_GateVtbl IGate::IGate_vtbl {
    &IGate::Destroy,
    &IGate::Update
};
IGate::IGate() : PLUGIN_Gate{ .vtbl = &IGate_vtbl } {
}

Plugin writers would implement their gates by inheriting from IGate, and then pass along the object by calling handle() and passing that value to the main app. Doesn't have to be C++ even, as long as PLUGIN_Gate is properly maintained.

// Main application side C++ wrapper
class Gate {
public:
    explicit Gate(PLUGIN_Gate* intf) : m_intf(intf) {
    }

    // Assuming we want an owning wrapper
    ~Gate() {
        m_intf->vtbl->Destroy(m_intf);
    }

    void update() {
        m_intf->vtbl->Update(m_intf);
    }

private:
    PLUGIN_Gate* m_intf;
};

Finally, the main application could use this type of wrapper to abstract away the C interface and don't deal with manual lifetime management and call chains.

Of course, your IGate is a trivial example. This can get complicated quick: you can't pass C++ types through the boundary (definitely no std::strings), multiple interfaces for the same objects would get very tricky very fast (single interface inheritance can be implemented though) etc etc. This is how we end up with things like Windows' COM