NSTextView selection highlights all characters even paragraph indents

Can't find any clue how to manage this.

By default, NSTextView selection highlights the whole size of its text container. It ignores line spacing, head or tail indents etc. But in Pages app selection doesn't highlight those ancillary parts, it highlight characters ONLY. And it highlights all the height of the line even if text container's height is smaller (paragraph spacing before and after).

I want to implement that behavior but can't understand where to begin. I've searched here, I've searched Apple docs, I've tried sample projects. Nothing.

Maybe someone can guide me in the right direction? Thanks!

I found that hamstergene's answer isn't correct. In fact, NSTextView highlights its text container bounds line by line.

So, if you use paragraph's head indents then the paragraph leading empty space will be highlighted. And if you select EOL character then the trailing space of the paragraph will be highlighted as well.

My solution was to nullify head and tail indents of the paragraph style (I cache them in the private variable and put them back when my text storage is accessed for printing) and simply adjust frame of the text container line via overrided lineFragmentRectForProposedRect: atIndex: writingDirection: remainingRect method of my NSTextContainer subclass.

But then I found much proper way. Just override func fillBackgroundRectArray(_ rectArray: UnsafePointer<NSRect>, count rectCount: Int, forCharacterRange charRange: NSRange, color: NSColor) of the NSLayoutManager, calculate your rects and call super with those rects. And if you calculated selection rectangles properly, you'll get the exact selection behavior like in Apple Pages or MS Word.

Simple and easy!

UPDATE Here's my code for calculating selection rects:

public override func fillBackgroundRectArray(_ rectArray: UnsafePointer<CGRect>, count rectCount: Int, forCharacterRange charRange: NSRange, color: OSColor) {
    
    // if characters are selected, make sure that we draw selection of those characters only, not the whole text container bounds
    guard let textView = textContainer(forCharacterIndex: charRange.location)?.textView,
        NSIntersectionRange(textView.selectedRange(), charRange).length > 0,
        let textStorage = self.textStorage as? ParagraphTextStorage else {
        super.fillBackgroundRectArray(rectArray, count: rectCount, forCharacterRange: charRange, color: color)
        return
    }
    
    let selectedGlyphRange = self.glyphRange(forCharacterRange: charRange, actualCharacterRange: nil)
    var selectionRectArray: [CGRect] = []

    enumerateLineFragments(forGlyphRange: selectedGlyphRange) { (rect, usedRect, textContainer, glyphRange, stop) in
        let lineCharRange = self.characterRange(forGlyphRange: glyphRange, actualGlyphRange: nil)
        let intersection = NSIntersectionRange(charRange, lineCharRange)
        
        // if selected all characters of the line, then we already have teir layout rects
        if intersection == lineCharRange {
            let paragraphIndex = textStorage.paragraphIndex(at: intersection.location)
            let paragraphRange = textStorage.paragraphRanges[paragraphIndex]
            
            let hasNewLineChar = lineCharRange.max == paragraphRange.max && paragraphRange.max < textStorage.length ||
                paragraphRange.max == lineCharRange.max && intersection.max == textStorage.length && paragraphIndex < textStorage.paragraphRanges.count - 1

            let newLineCharSize = hasNewLineChar ? self.newLineCharSize : .zero

            let lineRect = CGRect(x: usedRect.origin.x + textView.textContainerInset.width + textContainer.lineFragmentPadding,
                                  y: usedRect.origin.y + textView.textContainerInset.height - (rect.height - usedRect.height),
                                  width: usedRect.width + newLineCharSize.width - textContainer.lineFragmentPadding * 2,
                                  height: rect.height)
            selectionRectArray.append(lineRect)
        } else {
            // calculate rect for partially selected characters of the line
            let partialRect = self.usedLineRect(forCharacterRange: intersection, in: textContainer)
            selectionRectArray.append(partialRect)
        }
    }
    super.fillBackgroundRectArray(selectionRectArray, count: selectionRectArray.count, forCharacterRange: charRange, color: color)
}

public func usedLineRect(forCharacterRange charRange: NSRange, in textContainer: NSTextContainer) -> CGRect {
    guard let textView = textContainer.textView, let textStorage = textStorage as? ParagraphTextStorage else { return .zero }
            
    let glyphRange = self.glyphRange(forCharacterRange: charRange, actualCharacterRange: nil)
    let textContainer = self.textContainer(forGlyphAt: glyphRange.location, effectiveRange: nil) as! ModernTextContainer
    
    let paragraphIndex = textStorage.paragraphIndex(at: charRange.location)
    let paragraphRange = textStorage.paragraphRanges[paragraphIndex]
    let hasNewLine = paragraphRange.max == charRange.max && charRange.max < textStorage.length ||
        paragraphRange.max == charRange.max && charRange.max == textStorage.length && paragraphIndex < textStorage.paragraphRanges.count - 1
    let newLineCharSize = hasNewLine ? self.newLineCharSize : .zero

    // if new line is in range, boundingRect will return the whole width of the text container, fix that
    let noNewLineGlyphRange = hasNewLine ? NSRange(location: glyphRange.location, length: glyphRange.length - 1) : glyphRange
    
    let charRect = boundingRect(forGlyphRange: noNewLineGlyphRange, in: textContainer)
    let lineRect = lineFragmentRect(forGlyphAt: noNewLineGlyphRange.location, effectiveRange: nil, withoutAdditionalLayout: true)
    
    #if os(macOS)
    // respect the flipped coordinate system with abs function
    let rect = CGRect(x: charRect.origin.x + textView.textContainerInset.width,
                      y: abs(charRect.origin.y + textView.textContainerInset.height - (lineRect.height - charRect.height)),
                      width: charRect.width + newLineCharSize.width,
                      height: lineRect.height)
    #else
    let rect = CGRect(x: charRect.origin.x + textView.textContainerInset.left,
                      y: abs(charRect.origin.y + textView.textContainerInset.top - (lineRect.height - charRect.height)),
                      width: charRect.width + newLineCharSize.width,
                      height: lineRect.height)
    #endif
    
    return rect
}

The important part of this extremely fast-performed calculation is that I'm using my own ParagraphTextStorage implementation. Its purpose is to calculate paragraph ranges in real-time, right when the text storage is being edited. Knowing the correct paragraph ranges allows me to work with simple integers (as NSRange's) while calculating the selected rect. Otherwise I'd had to do a bunch of substrings to get to know whether the new line characters are selected or not. And those operations are really slow.

The implementation of my ParagraphTextStorage is here: https://github.com/CineDev/ParagraphTextKit