Hide Markdown Characters with NSLayoutManager in Swift

I am working on a rich text editor in a Mac app that uses Markdown syntax. I use NSTextStorage to watch for matches in Markdown syntax, then apply styles to the NSAttributedString in real time like this:

At this point, I'm already in way over my head on this stuff, but I'm excited to be making progress. :) This tutorial was very helpful.

As a next step, I want to hide the Markdown characters when the NSTextView's string is rendered. So in the example above, once the last asterisk is typed, I want the * * characters to be hidden and just see sample in bold.

I'm using an NSLayoutManager delegate and I can see the matched string, but I'm unclear on how to generate the modified glyphs/properties using the shouldGenerateGlyphs method. Here what I have so far:

func layoutManager(_: NSLayoutManager, shouldGenerateGlyphs _: UnsafePointer<CGGlyph>, properties _: UnsafePointer<NSLayoutManager.GlyphProperty>, characterIndexes _: UnsafePointer<Int>, font _: NSFont, forGlyphRange glyphRange: NSRange) -> Int {
    let pattern = "(\\*\\w+(\\s\\w+)*\\*)" // Look for stuff like *this*
    do {
        let regex = try NSRegularExpression(pattern: pattern)
        regex.enumerateMatches(in: textView.string, range: glyphRange) {
            match, _, _ in
            // apply the style
            if let matchRange = match?.range(at: 1) {
                print(matchRange) <!-- This is the range of *sample*

                // I am confused on how to provide the updated properties below...
                // let newProps = NSLayoutManager.GlyphProperty.null
                // layoutManager.setGlyphs(glyphs, properties: newProps, characterIndexes: charIndexes, font: aFont, forGlyphRange: glyphRange)
                // return glyphRange.length
            }
        }
    } catch {
        print("\(error.localizedDescription)")
    }

    return 0
}

How do I modify the stuff to pass into setGlyphs based on the range of the text I've found to hide the asterisks?

2022 Disclaimer

While I had some good results running this piece of code when I originally submitted this answer, another SO user (Tim S.) warned me that in some cases applying the .null glyph properties to some glyphs make cause the app the hang or crash.

From what I could gather this only happens with the .null property, and around glyph 8192 (2^13)... I have no idea why, and honestly it looks like a TextKit bug (or at least not something the TextKit engineers did expect the framework to be used for).

For modern apps, I'd advise you to take a look a TextKit 2, which is supposed to abstract away glyphs handling and simplify all that stuff (disclaimer in the disclaimer : I haven't tried it yet).

---

Foreword

I implemented this method to achieve something similar in my app. Keep in mind that this API is very poorly documented, so my solution is based on trial and error instead of a deep understanding of all the moving parts here.

In short: it should work but use at you own risk :)

Note also that I went into a lot of details in this answer in the hope to make it accessible to any Swift developer, even one without a background in Objective-C or C. You probably already know some of the things detailed hereafter.

On TextKit and Glyphs

One of the things that is important to understand is that a glyph is the visual representation of one or more characters, as explained in WWDC 2018 Session 221 "TextKit Best Practices" :

I'd recommend watching the whole talk. It's not super helpful in the particular case of understanding how layoutManager(_:shouldGenerateGlyphs:properties:characterIndexes:font:forGlyphRange:) works, but it gives a good amount of info on how TextKit works in general.

2023 update : I recommend also reading The Absolute Minimum Every Software Developer Must Know About Unicode in 2023 by Niki Tonsky

Understanding shouldGenerateGlyphs

So. From what I understand, each time NSLayoutManager is about to generate a new glyph before rendering them, it will give you a chance to modify this glyph by calling layoutManager(_:shouldGenerateGlyphs:properties:characterIndexes:font:forGlyphRange:).

Modifying Glyphs

As per the doc, if you want to modify the glyphs you should do so in this method by calling setGlyphs(_:properties:characterIndexes:font:forGlyphRange:).

Lucky for us, setGlyphs expects the exact same arguments as passed to us in shouldGenerateGlyphs. This means that in theory you could implement shouldGenerateGlyphs with just a call to setGlyphs and all would be well (but that wouldn't be super useful).

Return Value

The doc also says that the return value of shouldGenerateGlyphs should be "The actual glyph range stored in this method". It doesn't make much sense, as the expected return type is Int and not NSRange as one might expect. From trial and error, I think the framework expects us here to return the number of modified glyphs in the passed glyphRange, starting at index 0 (more on that later).

Also, "glyph range stored in this method" refers the call to setGlyphs, which will store the newly generated glyphs internally (imo this is very poorly worded).

A Not So Useful Implementation

So here's a correct implementation of shouldGenerateGlyphs (which... does nothing):

func layoutManager(_ layoutManager: NSLayoutManager, shouldGenerateGlyphs glyphs: UnsafePointer<CGGlyph>, properties: UnsafePointer<NSLayoutManager.GlyphProperty>, characterIndexes: UnsafePointer<Int>, font: UIFont, forGlyphRange glyphRange: NSRange) -> Int {
    layoutManager.setGlyphs(glyphs, properties: fixedPropertiesPointer, characterIndexes: characterIndexes, font: font, forGlyphRange: glyphRange)

    return glyphRange.length
}

it should also be equivalent to just returning 0 from the method:

By returning 0, it can indicate for the layout manager to do the default processing.

Doing Something Useful

So now, how can we edit our glyphs properties to make this method do something useful (like hiding glyphs)?

Accessing the Arguments Values

Most of the arguments of shouldGenerateGlyphs are UnsafePointer. That's the TextKit C API leaking in the Swift layer, and one of the things that make implementing this method a hassle in the first place.

A key point is that all the arguments of type UnsafePointer here are arrays (in C, SomeType * — or its Swift equivalent UnsafePointer<SomeType> — is the how we represent an array), and those arrays are all of length glyphRange.length. That's indirectly documented in the setGlyphs method:

Each array has glyphRange.length items

What this means is that with the nice UnsafePointer API Apple has given us, we can iterate on the elements of these array with a loop like this:

for i in 0 ..< glyphRange.length {
    print(properties[i])
}

Under the hood, UnsafePointer will do pointer arithmetic to access memory at the right address given any index passed to the subscript. I'd recommend reading the UnsafePointer documentation, this is really cool stuff.

Passing Something Useful to setGlyphs

We're now able to print the content of our arguments, and inspect what properties the framework's given us for each glyph. Now, how do we modify those and pass the result to setGlyphs?

First, it's important to note that while we could modify the properties argument directly, it's probably a bad idea, because that chunk of memory isn't owned by us and we have no idea what the framework will do with this memory once we exit the method.

So the right way to go about this is to create our own array of glyph properties, and then pass that to setGlyphs:

var modifiedGlyphProperties = [NSLayoutManager.GlyphProperty]()
for i in 0 ..< glyphRange.length {
    // This contains the default properties for the glyph at index i set by the framework.
    var glyphProperties = properties[i]
    // We add the property we want to the mix. GlyphProperty is an OptionSet, we can use `.insert()` to do that.
    glyphProperties.insert(.null)
    // Append this glyph properties to our properties array.
    modifiedGlyphProperties.append(glyphProperties)
}

// Convert our Swift array to the UnsafePointer `setGlyphs` expects.
modifiedGlyphProperties.withUnsafeBufferPointer { modifiedGlyphPropertiesBufferPointer in
    guard let modifiedGlyphPropertiesPointer = modifiedGlyphPropertiesBufferPointer.baseAddress else {
        fatalError("Could not get base address of modifiedGlyphProperties")
    }

    // Call setGlyphs with the modified array.
    layoutManager.setGlyphs(glyphs, properties: modifiedGlyphPropertiesPointer, characterIndexes: characterIndexes, font: font, forGlyphRange: glyphRange)
}

return glyphRange.length

It's important to read the original glyph properties from the properties array and adding your custom ones to this base value (with the .insert() method). Otherwise you'd overwrite the default properties of your glyphs and weird things would happen (I've seen \n characters not inserting a visual line break anymore for example).

Deciding Which Glyphs to Hide

The previous implementation should work fine, but right now we're unconditionally hiding all generated glyphs, and it would be much more useful if we could hide only some of them (in your case when the glyph is *).

Hiding Based on Characters Values

To do that, you'll probably need to access the characters used to generate the final glyph. However, the framework doesn't give you the characters but their index in the string for each generated glyph. You'll need to iterate over these indexes and look into your NSTextStorage to find the corresponding characters.

Unfortunately, this is not a trivial task: Foundation uses UTF-16 code units to represent strings internally (that's what NSString and NSAttributedString use under the hood). So what the framework gives us with characterIndexes is not the indexes of "characters" in the usual sense of the word, but the indexes of UTF-16 code units^†.

Most of the time, each UTF-16 code unit will be used to generate a unique glyph, but in some cases multiple code units will be used to generate a unique glyph (this is called a UTF-16 surrogate pair, and is common when handling string with emojis). I'd recommend testing your code with some more "exotic" strings like for example:

textView.text = "Officiellement nous (👨‍👩‍👧‍👧) vivons dans un cha\u{0302}teau 🏰 海"

So, to be able to compare our characters, we first need to convert them to a simple representation of what we usually mean by "character":

/// Returns the extended grapheme cluster at `index` in an UTF16View, merging a UTF-16 surrogate pair if needed.
private func characterFromUTF16CodeUnits(_ utf16CodeUnits: String.UTF16View, at index: Int) -> Character {
    let codeUnitIndex = utf16CodeUnits.index(utf16CodeUnits.startIndex, offsetBy: index)
    let codeUnit = utf16CodeUnits[codeUnitIndex]

    if UTF16.isLeadSurrogate(codeUnit) {
        let nextCodeUnit = utf16CodeUnits[utf16CodeUnits.index(after: codeUnitIndex)]
        let codeUnits = [codeUnit, nextCodeUnit]
        let str = String(utf16CodeUnits: codeUnits, count: 2)
        return Character(str)
    } else if UTF16.isTrailSurrogate(codeUnit) {
        let previousCodeUnit = utf16CodeUnits[utf16CodeUnits.index(before: codeUnitIndex)]
        let codeUnits = [previousCodeUnit, codeUnit]
        let str = String(utf16CodeUnits: codeUnits, count: 2)
        return Character(str)
    } else {
        let unicodeScalar = UnicodeScalar(codeUnit)!
        return Character(unicodeScalar)
    }
}

Then we can use this function to extract the characters from our textStorage, and test them:

// First, make sure we'll be able to access the NSTextStorage.
guard let textStorage = layoutManager.textStorage else {
    fatalError("No textStorage was associated to this layoutManager")
}


// Access the characters.
let utf16CodeUnits = textStorage.string.utf16
var modifiedGlyphProperties = [NSLayoutManager.GlyphProperty]()
for i in 0 ..< glyphRange.length {
    var glyphProperties = properties[i]
    let character = characterFromUTF16CodeUnits(utf16CodeUnits, at: characterIndex)

    // Do something with `character`, e.g.:
    if character == "*" {
        glyphProperties.insert(.null)
    }
    
    modifiedGlyphProperties.append(glyphProperties)
}
    
// Convert our Swift array to the UnsafePointer `setGlyphs` expects.
modifiedGlyphProperties.withUnsafeBufferPointer { modifiedGlyphPropertiesBufferPointer in
    guard let modifiedGlyphPropertiesPointer = modifiedGlyphPropertiesBufferPointer.baseAddress else {
        fatalError("Could not get base address of modifiedGlyphProperties")
    }

    // Call setGlyphs with the modified array.
    layoutManager.setGlyphs(glyphs, properties: modifiedGlyphPropertiesPointer, characterIndexes: characterIndexes, font: font, forGlyphRange: glyphRange)
}

return glyphRange.length

Note that in the case of surrogate pairs, the loop will be executed twice (once on the lead surrogate, and once on the trail surrogate), and you'll end up comparing the same resulting character twice. This is fine though as you need to apply the same modification you want on both "parts" of the generated glyph.

Hiding Based on the TextStorage String Attributes

That's not what you've asked for in your question, but for completion's sake (and because it's what I do in my app), here how you can access your textStorage string attributes to hide some glyphs (in this example I'll hide all the parts of the text with an hypertext link):

// First, make sure we'll be able to access the NSTextStorage.
guard let textStorage = layoutManager.textStorage else {
    fatalError("No textStorage was associated to this layoutManager")
}

// Get the first and last characters indexes for this glyph range,
// and from that create the characters indexes range.
let firstCharIndex = characterIndexes[0]
let lastCharIndex = characterIndexes[glyphRange.length - 1]
let charactersRange = NSRange(location: firstCharIndex, length: lastCharIndex - firstCharIndex + 1)

var hiddenRanges = [NSRange]()
textStorage.enumerateAttributes(in: charactersRange, options: []) { attributes, range, _ in
    for attribute in attributes where attribute.key == .link {
        hiddenRanges.append(range)
    }
}

var modifiedGlyphProperties = [NSLayoutManager.GlyphProperty]()
for i in 0 ..< glyphRange.length {
    let characterIndex = characterIndexes[i]
    var glyphProperties = properties[i]

    let matchingHiddenRanges = hiddenRanges.filter { NSLocationInRange(characterIndex, $0) }
    if !matchingHiddenRanges.isEmpty {
        glyphProperties.insert(.null)
    }

    modifiedGlyphProperties.append(glyphProperties)
}

// Convert our Swift array to the UnsafePointer `setGlyphs` expects.
modifiedGlyphProperties.withUnsafeBufferPointer { modifiedGlyphPropertiesBufferPointer in
    guard let modifiedGlyphPropertiesPointer = modifiedGlyphPropertiesBufferPointer.baseAddress else {
        fatalError("Could not get base address of modifiedGlyphProperties")
    }

    // Call setGlyphs with the modified array.
    layoutManager.setGlyphs(glyphs, properties: modifiedGlyphPropertiesPointer, characterIndexes: characterIndexes, font: font, forGlyphRange: glyphRange)
}

return glyphRange.length

---

^† To understand the differences between those, I'd recommend reading the Swift Documentation on "Strings and Characters". Note also that what the framework calls "character" here is not the same as what Swift calls a Character (or "Extended Grapheme Clusters"). Again, "character" for the TextKit framework is an UTF-16 code unit (represented in Swift by Unicode.UTF16.CodeUnit).

---

Update 2020-04-16: Make use of .withUnsafeBufferPointer to convert the modifiedGlyphProperties array to an UnsafePointer. It removes the need to have an instance variable of the array to keep it alive in memory.