My goal is to make text selection behaviour similar to Apple Pages, MS Word and any other text processor, when NSTextView selects ONLY text glyphs and not any indents or the whole NSTextContainer size like by default NSTextView's behavior.
With that goal in mind, I subclassed NSLayoutManager and overrided func rectArray(forCharacterRange charRange: NSRange, withinSelectedCharacterRange selCharRange: NSRange, in container: NSTextContainer, rectCount: UnsafeMutablePointer<Int>) -> NSRectArray?
In my override I simply calculated new rects that bound just glyphs and rejects any indents, blank spaces etc.
Everything worked fine. But suddenly in macOS 10.11 Apple did deprecate the method I'm using. And suggested to use func enumerateEnclosingRectsForGlyphRange: withinSelectedGlyphRange: inTextContainer: usingBlock: instead.
But it seems that this new method doesn't work in a way the previous deprecated method did. It's impossible to customize enclosing rects any more. All I can do is to enumerate through rects TextKit did calculate. And now it's impossible to achieve desirable text selection behaviour because the rects are read-only. And, by the way, the new method doesn't get called at all by the TextKit. So there's no point for me to think about it.
What should I do?
While the surprise with nonequivalent method replacement remains, I was able to find a solution.
It turned out that I don't need rectArrayForCharacterRange method to do the job. I just overrided func fillBackgroundRectArray(_ rectArray: UnsafePointer<NSRect>, count rectCount: Int, forCharacterRange charRange: NSRange, color: NSColor) and put my rect calculations in there.
Everything works like a charm now!