How to define an abstract (opaque) interface for a polymorphic object type?

ReasonML

module type T = {
  type t('a); // Does not work
  type b; // Works
};

module A: T = {
  type t('a) = {.. b: bool} as 'a;
  type b = bool;
};
module B: T = {
  type t('a) = {.. c: int} as 'a;
  type b = int;
};

Ocaml

module type T  = sig
  type 'a t /* Doesn't work */
  type b /* Works */
end
module A : T = struct type 'a t = < b :bool  ;.. > as 'a
                      type b = bool end 
module B : T = struct type 'a t = < c :int  ;.. > as 'a
                      type b = int end  

How can I define the module type A t('a) so that it is abstract but compatible with the open polymorphic object types in the implementations?

Types <b : bool; .. > and <c : int; ..> are not compatible much like int and bool are not compatible. In other words, if we will put row polymorphism aside, and focus on simple type constructors, then you're trying to define an interface that matches type int and bool and nothing else, aka bounded polymorphism.

It is also important to understand that subtyping is not inheritance. In your case, you have two classes of objects, the b-class objects that have method b

class virtual b = object
  method virtual b : bool
end

and the c- class of objects that have method c,

class virtual c = object
  method virtual c : int
end

and we can define a class of objects bc that have both methods, naturally via inheritance,

class virtual bc = object
  inherit b
  inherit c
end

Now, let's cooks some objects to play with,

let b : b = object method b = true end
let c : c = object method c = 42 end
let bc : bc = object
  method b = false
  method c = 56
end

We can see that despite that bc class type is defined as inherited from b and c we can't coerce b to c,

# (b : b :> bc);;
Line 1, characters 0-13:
1 | (b : b :> bc);;
    ^^^^^^^^^^^^^
Error: Type b = < b : bool > is not a subtype of bc = < b : bool; c : int > 
       The first object type has no method c

and this makes perfect sense since we're trying to downcast an object of the base class to an object of the derived class, which is an illegal operation. Therefore, when you have a hierarchy of class types ordered by inheritance, the base classes are super types and the derived classes are subtypes. E.g., if v inherits from u then v is a subtype of u,

v inherits-from u
-----------------
   u :> v

Once you get a clear understanding of this, you can design a proper interface.