How can a class be defined in such way that it inherits from QObject as well within a QObject-derived class?

In class A, I define a class B. I want to hide the implementation of class B. The class B inherits from QObject.

// a.h
class A : public QObject
{
    Q_OBJECT
public:
    explicit A(QObjcet* parent=nullptr);
private:
    class B;
    QSharedPointer<B> m_impl;
};

//a.cpp
class A::B : public QObject
{
    Q_OBJECT
public:
    ...
};

A::A(QObject* parent) : 
    QObject(parent),
    m_impl(QSharedPointer<B>(new B()))
{
}

But it has errors:

undefined reference to 'vtable for A::B'
undefined reference to 'A::B::staticMetaObject'

What causes them, and how can I fix them?

For QObject-derived class to be compiled you need an extra unit containing definitions for Q_OBJECT macro, generated by meta-object compiler(moc) utility based on conent of the header where class is declared.

Nested classes or classes inherited from multiple QObjects are not supported by Qt and may or may not work. Meta-object compiler also isn't designed to be used on .cpp files because of its limitted ability to parse code. By default Qt toolchains do not run moc on .cpp files.

You can create an additional compilation step to run moc but that's a dirty and oddball solution, you'd better to have a separate, "private" header if your class is derived from QObject, hidden in a folder no foreign unit would look up.

//impl/a_private.h
class A::B : public QObject
{
    Q_OBJECT
public:
    ...
};

 //a.cpp
 #include "a.h"
 #include "impl/a_private.h"
 ...

This looks like B is meant to be a private ("bridged") class used within A:

class B;
QSharedPointer<B> m_impl;

In that case you must use Qt's PIMPL aka Cheshire cat pattern, which is a variant of Bridge pattern.

How to use the Qt's PIMPL idiom?

TLDR: A::B would become a class APrivate in global namespace