For example:
class B;
class A {
public:
B f();
};
int main(int, char**) {
A a; // no exception and error
return 0;
}
The class A may be an incomplete type.
Why can class A be instantiated in this example?
I know that the program will fail to compile when the member function f is called in the code.
In C++ entities don't need to be defined in the same translation unit in which they are used (exceptions apply). So it not being defined in one translation unit doesn't mean that you won't define and explicitly instantiate the member function in another translation unit.
Also, it is not necessary to define entities that are not ODR-used (e.g. called) at all in C++.
This is exactly the same for free functions. You also only need to define a free function if you actually ODR-use (e.g. call) it and the definition needs to be only in one translation unit.
The class A may be an incomplete type.
A is complete after the class A { /*...*/ }; definition. Whether member functions are defined doesn't affect completeness.
I know that the program will fail to compile when the member function f is called in the code.
But it will fail only in the linker step when all translation units are linked together, because before that the compiler can't know that the function isn't defined and instantiated somewhere else. If the function isn't ODR-used (e.g. called), then the linker has no reason to look for a definition for it anyway.
Note that B being incomplete is also not a problem. Return types need to be complete only in the definition of a function, not its declaration, and you can have B be completed before you define the member function in another translation unit.
B also needs to be complete when calling the function, however not in general for other ODR-uses. This applies to all return and parameter types. Still, B can be completed without defining the function. That can still happen in another translation unit.
ODR in the above means "one definition rule". This is the rule that there shall be exactly one definition of a given entity in the whole program. The rule doesn't apply in full to all entities and in particular templated entities can be defined once in each translation as long as the definitions are identical (exact rules are more complicated).
"ODR-use" refers to a use of an entity that would trigger the one definition rule to require (at least) one definition of the entity to exist somewhere in the program. These are for example calling a function or taking the address of a function.
See https://en.cppreference.com/w/cpp/language/definition for details.