NSLayoutManager hides new line characters no matter what I do

I'm trying to show invisible characters like the new line character in my NSTextView subclass. The usual approach like overriding drawGlyph method of NSLayoutManager is a bad idea because it's too slow and not work properly with multi-paged layout.

What I'm trying to do is to override the setGlyph method of the NSLayoutManager so it would replace invisible "\n" glyph with "¶" glyph and " " with "∙".

And it works on the " " space glyphs but has no effect on the new line characters.

public override func setGlyphs(_ glyphs: UnsafePointer<CGGlyph>, properties props: UnsafePointer<NSGlyphProperty>, characterIndexes charIndexes: UnsafePointer<Int>, font aFont: Font, forGlyphRange glyphRange: NSRange) {
    var substring = (self.currentTextStorage.string as NSString).substring(with: glyphRange)

    // replace invisible characters with visible
    if PreferencesManager.shared.shouldShowInvisibles == true {
        substring = substring.replacingOccurrences(of: " ", with: "\u{00B7}")
        substring = substring.replacingOccurrences(of: "\n", with: "u{00B6}")
    }

    // create a CFString
    let stringRef = substring as CFString
    let count = CFStringGetLength(stringRef)

    // convert processed string to the C-pointer
    let cfRange = CFRangeMake(0, count)
    let fontRef = CTFontCreateWithName(aFont.fontName as CFString?, aFont.pointSize, nil)
    let characters = UnsafeMutablePointer<UniChar>.allocate(capacity: MemoryLayout<UniChar>.size * count)
    CFStringGetCharacters(stringRef, cfRange, characters)

    // get glyphs for the pointer of characters
    let glyphsRef = UnsafeMutablePointer<CGGlyph>.allocate(capacity: MemoryLayout<CGGlyph>.size * count)
    CTFontGetGlyphsForCharacters(fontRef, characters, glyphsRef, count)

    // set those glyphs
    super.setGlyphs(glyphsRef, properties:props, characterIndexes: charIndexes, font: aFont, forGlyphRange: glyphRange)
}

Then I came up with an idea: it looks like NSTypesetter marks new line char ranges like those it shouldn't process at all. So I subclassed NSTypesetter and did override a method:

override func setNotShownAttribute(_ flag: Bool, forGlyphRange glyphRange: NSRange) {
    let theFlag = PreferencesManager.shared.shouldShowInvisibles == true ? false : true
    super.setNotShownAttribute(theFlag, forGlyphRange: glyphRange)
}

But it's not working. NSLayoutManager still won't generate a glyph for the new line character, no matter what glyph I create.

What am I doing wrong?

As I figured out, the default implementation of NSTypesetter's setNotShownAttribute: of the class doesn't change already generated glyphs in its glyph storage. So, call of super doesn't produce any effect. I just have to replace glyphs manually before calling super.

So, the most efficient implementation of showing invisible characters (you will see the difference while zooming the view) is this:

Limitations of this approach: if your app has to have multiple fonts in text view, then this approach might not be such a good idea, because the font of those displayed invisible characters will be different as well. And that's not what you might want to achieve.

1. Subclass NSLayoutManager and override setGlyphs to show space chars:

   public override func setGlyphs(_ glyphs: UnsafePointer<CGGlyph>, properties props: UnsafePointer<NSGlyphProperty>, characterIndexes charIndexes: UnsafePointer<Int>, font aFont: Font, forGlyphRange glyphRange: NSRange) {
       var substring = (self.currentTextStorage.string as NSString).substring(with: glyphRange)
   
       // replace invisible characters with visible
       if PreferencesManager.shared.shouldShowInvisibles == true {
           substring = substring.replacingOccurrences(of: " ", with: "\u{00B7}")
       }
   
       // create a CFString
       let stringRef = substring as CFString
       let count = CFStringGetLength(stringRef)
   
       // convert processed string to the C-pointer
       let cfRange = CFRangeMake(0, count)
       let fontRef = CTFontCreateWithName(aFont.fontName as CFString?, aFont.pointSize, nil)
       let characters = UnsafeMutablePointer<UniChar>.allocate(capacity: MemoryLayout<UniChar>.size * count)
       CFStringGetCharacters(stringRef, cfRange, characters)
   
       // get glyphs for the pointer of characters
       let glyphsRef = UnsafeMutablePointer<CGGlyph>.allocate(capacity: MemoryLayout<CGGlyph>.size * count)
       CTFontGetGlyphsForCharacters(fontRef, characters, glyphsRef, count)
   
       // set those glyphs
       super.setGlyphs(glyphsRef, properties:props, characterIndexes: charIndexes, font: aFont, forGlyphRange: glyphRange)
   }
   

2. Subclass NSATSTypesetter and assign it to your NSLayoutManager subclas. The subclass will display the new line characters and make sure that every invisible character will be drawn with a different color:

   class CustomTypesetter: NSATSTypesetter {
   
       override func setNotShownAttribute(_ flag: Bool, forGlyphRange glyphRange: NSRange) {
           var theFlag = flag
   
           if PreferencesManager.shared.shouldShowInvisibles == true   {
               theFlag = false
   
               // add new line glyphs into the glyph storage
               var newLineGlyph = yourFont.glyph(withName: "paragraph")
               self.substituteGlyphs(in: glyphRange, withGlyphs: &newLineGlyph)
   
               // draw new line char with different color
               self.layoutManager?.addTemporaryAttribute(NSForegroundColorAttributeName, value: NSColor.invisibleTextColor, forCharacterRange: glyphRange)
           }
   
           super.setNotShownAttribute(theFlag, forGlyphRange: glyphRange)
       }
   
       /// Currently hadn't found any faster way to draw space glyphs with different color
       override func setParagraphGlyphRange(_ paragraphRange: NSRange, separatorGlyphRange paragraphSeparatorRange: NSRange) {
           super.setParagraphGlyphRange(paragraphRange, separatorGlyphRange: paragraphSeparatorRange)
   
           guard PreferencesManager.shared.shouldShowInvisibles == true else { return }
   
           if let substring = (self.layoutManager?.textStorage?.string as NSString?)?.substring(with: paragraphRange) {
               let expression = try? NSRegularExpression.init(pattern: "\\s", options: NSRegularExpression.Options.useUnicodeWordBoundaries)
               let sunstringRange = NSRange(location: 0, length: substring.characters.count)
   
               if let matches = expression?.matches(in: substring, options: NSRegularExpression.MatchingOptions.withoutAnchoringBounds, range: sunstringRange) {
                   for match in matches {
                       let globalSubRange = NSRange(location: paragraphRange.location + match.range.location, length: 1)
                       self.layoutManager?.addTemporaryAttribute(NSForegroundColorAttributeName, value: Color.invisibleText, forCharacterRange: globalSubRange)
                   }
               }
           }
       }
   }
   

3. To show/hide invisible characters just call:

   let storageRange = NSRange(location: 0, length: currentTextStorage.length)
   layoutManager.invalidateGlyphs(forCharacterRange: storageRange, changeInLength: 0, actualCharacterRange: nil)
   layoutManager.ensureGlyphs(forGlyphRange: storageRange)