How to define a custom NSUnderlineStyle

Looking at the documentation for NSLayoutManager, specifically the drawUnderlineForGlyphRange:underlineType:baselineOffset:lineFragmentRect:lineFragmentGlyphRange:containerOrigin: method, I noticed the following (emphasis mine):

underlineVal
The style of underlining to draw. This value is a mask derived from the value for NSUnderlineStyleAttributeName—for example, (NSUnderlinePatternDash | NSUnderlineStyleThick). Subclasses can define custom underlining styles.

My question is: how exactly is that meant to be done?

NSUnderlineStyle is an enum, which you cannot extend or override. You can of course provide a random raw Int value for the attribute, not covered by the enum cases:

self.addAttribute(NSUnderlineStyleAttributeName, value: 100022, range: lastUpdatedWordRange)

Which will deliver an "invalid" but usable underlineType to the Layout Manger:

But this hardly feels safe and is definitely inelegant.

I was not able to find any examples online or further clues in Apple documentation on what those mythical custom underline style types look like. I'd love to know if I'm missing something obvious.

I have an example project here that I used for a talk on TextKit that I gave a while back that does exactly what you're looking for: https://github.com/dtweston/text-kit-example

The underline in this case is a squiggly line:

The meat of the solution is a custom NSLayoutManager:

let CustomUnderlineStyle = 0x11

class UnderlineLayoutManager: NSLayoutManager {
    
    func drawFancyUnderlineForRect(_ rect: CGRect) {
        let left = rect.minX
        let bottom = rect.maxY
        let width = rect.width
        
        let path = UIBezierPath()
        path.move(to: CGPoint(x: left, y: bottom))
        var x = left
        var y = bottom
        var i = 0
        while (x <= left + width) {
            path.addLine(to: CGPoint(x: x, y: y))
            x += 2
            if i % 2 == 0 {
                y = bottom + 2.0
            }
            else {
                y = bottom
            }
            i += 1;
        }
        
        path.stroke()
    }
    
    override func drawUnderline(forGlyphRange glyphRange: NSRange, underlineType underlineVal: NSUnderlineStyle, baselineOffset: CGFloat, lineFragmentRect lineRect: CGRect, lineFragmentGlyphRange lineGlyphRange: NSRange, containerOrigin: CGPoint) {
        
        if underlineVal.rawValue & CustomUnderlineStyle == CustomUnderlineStyle {
            
            let charRange = characterRange(forGlyphRange: glyphRange, actualGlyphRange: nil)
            if let underlineColor = textStorage?.attribute(NSUnderlineColorAttributeName, at: charRange.location, effectiveRange: nil) as? UIColor {
                underlineColor.setStroke()
            }
            
            if let container = textContainer(forGlyphAt: glyphRange.location, effectiveRange: nil) {
                let boundingRect = self.boundingRect(forGlyphRange: glyphRange, in: container)
                let offsetRect = boundingRect.offsetBy(dx: containerOrigin.x, dy: containerOrigin.y)
                
                drawFancyUnderlineForRect(offsetRect)
            }
        }
        else {
            super.drawUnderline(forGlyphRange: glyphRange, underlineType: underlineVal, baselineOffset: baselineOffset, lineFragmentRect: lineRect, lineFragmentGlyphRange: lineGlyphRange, containerOrigin: containerOrigin)
        }
    }
}