Expand UITextView and then scroll after certain point

I have been struggling to set create this, I have read numerous posts on here, but do not understand what I am doing wrong.

I have a UITextView.

I would like it to expand in height, up to a number and then fix at that height with scrolling enabled.

I have set my UITextFieldDelegateand added the following -

    func textViewDidChange(_ textView: UITextView) {
        let newSize = textView.sizeThatFits(CGSize(width: 0, height: CGFloat.greatestFiniteMagnitude))
        textView.frame.size = CGSize(width: 0, height: min(100, newSize.height))
    }

Doing this allow the to keep growing.

I'd now like it to stop and scroll instead at for example, a height of 100.

    func textViewDidChange(_ textView: UITextView) {
        let newSize = textView.sizeThatFits(CGSize(width: 0, height: CGFloat.greatestFiniteMagnitude))

        let newHeight = min(100, newSize.height)

        textView.frame.size = CGSize(width: 0, height: newHeight)

        textView.isScrollEnabled = newHeight >= 100
    }

All that happens at this point is the scrollview shrinks when isScrollEnabled is set as true.

My textView is anchored only to top, leading and trailing of it's parent.

How can I force it to not collapse once scroll is enabled?

EDIT

I have added the following which works, however the text seems to 'wobble' when entering anything at the fixed height state

    func textViewDidChange(_ textView: UITextView) {
        let newSize = textView.sizeThatFits(CGSize(width: 0, height: CGFloat.greatestFiniteMagnitude))

        let newHeight = min(100, newSize.height)

        textView.frame.size = CGSize(width: 0, height: newHeight)

        if newHeight >= 100 {
            textView.isScrollEnabled = true
            textView.heightAnchor.constraint(equalToConstant: 100).isActive = true
        } else {
            textView.isScrollEnabled = false
            textView.heightAnchor.constraint(equalToConstant: 100).isActive = false
        }
    }

It's better to make a default height constraint and connect it's outlet and play with it's constant

self.txH.constant = newHeight
self.view.layoutIfNeeded()