How to Interact with Custom QT Rendered Object In QTextEdit

I'm trying to build a markdown based notes app using the QT Library in C++. I started out using the QTextEdit class and have added the new graphic (a checkbox) I want to render inline as a CustomObjectRenderer. I render the object like this into my the QTextEdit body:

// Render checkbox
if (blockText == "- [ ]") {
    cursor.beginEditBlock();
    cursor.removeSelectedText();

    QTextCharFormat fmt;
    fmt.setObjectType(QTextFormat::UserObject + 1);
    fmt.setProperty(1000, "checkbox");
    fmt.setProperty(1001, true);
    fmt.setFontPointSize(FONT_SIZE);
    cursor.insertText(QString(QChar::ObjectReplacementCharacter), fmt);
    cursor.endEditBlock();

    return;
}

I can't seem to figure out why with my mousesPressEvent function, it only unchecks/checks the box (custom rendered object) when I'm on the left side of the checkbox, but not on the right.

void MarkdownEditor::mousePressEvent(QMouseEvent *event)
{
    QTextCursor cursor = cursorForPosition(event->pos());
    cursor.movePosition(QTextCursor::Right, QTextCursor::KeepAnchor, 1); // Select one character
        
    QTextCharFormat fmt = cursor.charFormat();
        
    if (fmt.objectType() == QTextFormat::UserObject + 1 &&
        fmt.property(1000).toString() == "checkbox") {
        
        //Toggle the checkbox state
        bool checked = fmt.property(1001).toBool();
        fmt.setProperty(1001, !checked);
        
        cursor.setCharFormat(fmt); // <- this now applies to the character under the cursor
    
        return;
    }
        
    QTextEdit::mousePressEvent(event);
}

This is also the checkbox code (CustomObjectRenderer)

class CustomObjectRenderer : public QObject, public QTextObjectInterface {
    Q_OBJECT
    Q_INTERFACES(QTextObjectInterface)

public:
    explicit CustomObjectRenderer(QObject *parent = nullptr) : QObject(parent) {}

    QSizeF intrinsicSize(QTextDocument *, int, const QTextFormat &) override {
        return QSizeF(20, 20);  // Size of the checkbox
    }

    void drawObject(QPainter *painter, const QRectF &rect, QTextDocument *, int, const QTextFormat &format) override
    {
        if (format.property(1000).toString() != "checkbox") return;

        bool checked = format.property(1001).toBool();

        painter->save();

        const qreal radius = 2.0;
        const QColor backgroundColor = QColor("#111111");
        const QColor checkColor = QColor("#ffffff");

        painter->setRenderHint(QPainter::Antialiasing);

        // Define size of checkbox (same as in intrinsicSize)
        const QSizeF boxSize = QSizeF(20, 20);

        // Calculate centered rect within `rect`
        QRectF boxRect(
            rect.left(),
            rect.top() + (rect.height() - boxSize.height()) / 2.0 + 2,  // vertically center it
            boxSize.width(),
            boxSize.height()
            );

        painter->setBrush(backgroundColor);
        painter->setPen(Qt::NoPen); // or a border if desired
        painter->drawRoundedRect(boxRect, radius, radius);

        if (checked) {
            QPen pen(checkColor, 2);
            painter->setPen(pen);

            QPointF p1(boxRect.left() + boxRect.width() * 0.25, boxRect.top() + boxRect.height() * 0.55);
            QPointF p2(boxRect.center().x(), boxRect.bottom() - boxRect.height() * 0.3);
            QPointF p3(boxRect.right() - boxRect.width() * 0.2, boxRect.top() + boxRect.height() * 0.3);

            painter->drawLine(p1, p2);
            painter->drawLine(p2, p3);
        }

        painter->restore();
    }


};

Any pointers on how I can understand the underlying graphics rendering architecture of QT apps? I'm heavily relying on LLMs for this but any pointers to understand how this works at a deeper level so I can troubleshoot would be great.

Solution

QTextEdit's cursorForPosition() returns a QTextCursor with its position set from the result of the document layout's hitTest() function.

The main purpose of hitTest() is to get the closest appropriate ("legal") insertion index in the text based on a given point, and that original point is obviously almost never exactly at a valid insertion point: the most important reason is that the user will rarely be able (or care) to put the mouse precisely between characters. Therefore, if the user clicks anywhere but that precise point, that function will eventually decide if the intended position is before or after that character.

The computation that results in that "before" or "after" is actually relatively complex, but the general concept can be simplified as: if the x is in the left half of the character, return the character position, otherwise if it's on the right half, return the position after that.

Let's put aside the case of the user object for a moment, and consider a simple example: suppose you have a document with a default font having a big point size, and a "large" character as w at the beginning of that document, followed by other characters with different formats (bold, italic, color, etc.).

If you click on the left side of the "w", the text cursor will be placed at position 0, and calling cursor.movePosition(QTextCursor::Right, QTextCursor::KeepAnchor, 1) will cause the cursor to move on the right of the letter (position 1).
Since QTextCursor::charFormat() "[r]eturns the format of the character immediately before the cursor position()", you will properly get the format used for the "w" letter. In fact, if you only wanted to get the format, you wouldn't even need to use KeepAnchor, because the *Format() getter functions used by QTextCursor do not care the about selection at all, they are just interested in the current position().

Now let's click on the right half of that first "w": the QTextCursor will be positioned at 1 for the above reasons, but then you moveCursor() by one character on the right, therefore you will get the format of the character after that (position() is 2), which may not be the same of the "w" at the beginning.

The hitTest() function used for cursorForPosition() always uses the FuzzyHit accuracy argument, which is reasonable for the purpose of editing, in order to place the caret where the user probably intended, even when they click in a position where no text exists at all.

Using the ExactHit accuracy, instead, will always result in the cursor position before the given point (normally, the left of the character for left to right writing systems, even when clicking near its right edge), or an invalid (-1) position if no valid cursor position exists at that point of the document layout.

That version of the call is also the one used by the formatAt() and related imageAt() functions introduced since Qt 5.8.

In order to get the correct format, along with the selection used to update its properties, you therefore need to follow this procedure:

get a point mapped to the document (based on the pos() of the event), by adding the values of the horizontal and vertical scroll bars, no matter if they're hidden;
get the documentLayout() and call hitTest() with the mapped point and ExactHit arguments;
if it's less than 0, call the default mousePressEvent() and return;
create a QTextCursor for the document and set the position returned by hitTest();
if the cursor is atEnd(), use moveCursor() with PreviousCharacter, without keeping the anchor;
call moveCursor() with NextCharacter (which is more appropriate than Right, as the contents may have right-to-left text) and KeepAnchor;
check the charFormat() and do the rest if it matches your case;

^{Note that points 5 and 6 are valid and relevant even for a virtually empty document, especially if the cursor has some char format already set (for instance, when calling QTextEdit.setFontUnderline(True) on a new document): the benefit of moveCursor() (which internally uses QTextCursor::movePosition()) is that it's always a safe operation, as it never attempts to move the cursor position outside the document extent (less than 0, or greater than the document's characterCount()).}

I cannot provide a reliable C++ code as I only use Qt from Python, but you can consider the following as a pseudo-code implementation of the above procedure:

class MarkdownEditor(QTextEdit):
    ...
    def mousePressEvent(self, event):
        offset = QPoint(
            self.horizontalScrollBar().value(),
            self.verticalScrollBar().value()
        )
        curPos = self.document().documentLayout().hitTest(
            event.pos() + offset, Qt.ExactHit)
        if curPos >= 0:
            tc = QTextCursor(self.document())
            tc.setPosition(curPos)
            if tc.atEnd():
                tc.movePosition(QTextCursor.PreviousCharacter)
            tc.movePosition(QTextCursor.NextCharacter, QTextCursor.KeepAnchor)
            fmt = cursor.charFormat()
            if (
                fmt.objectType() == QTextFormat.UserObject + 1
                and fmt.property(1000) == 'checkbox'
            ):
                fmt.setProperty(1001, not bool(fmt.property(1001)))
                tc.setCharFormat(fmt)
                return
        super().mousePressEvent(event)

^{For Qt6 the mouse position is event.position(), and offset should be a QPointF in order to properly use the + operator.}