Multiple CGAffineTranform statements dont give smooth results

I'm applying a rotation and scaling on my UIView. When I apply ONE of these transform, it works fine but when I try to apply both the result is kind of not so smooth. It appears as it resets and the starts applying. Here's my code

var location:CGPoint = CGPoint.zero
var rotatingLoaction:CGPoint = CGPoint.zero
var deltaAngle: CGFloat!
func handleRotateAndResizeWithPanGestureRecognizer(_ recognizer: UIPanGestureRecognizer) -> Void
{
    location = recognizer.location(in: self.superview)

    let sizeCenter = CGPoint(x: (self.frame.origin.x + self.frame.size.width) / 2.0, y: (self.frame.origin.y + self.frame.size.height) / 2.0)
    
    if recognizer.state == .began {
        initialDistance = sizeCenter.getDistance(point: location)
        deltaAngle = atan2(location.y - sizeCenter.y, location.x - sizeCenter.x) - self.transform.getAngle()
    }
    else if recognizer.state == .changed
    {
        let ang = atan2(location.y - sizeCenter.y,
                          location.x - sizeCenter.x);
        let angleDiff = deltaAngle - ang;
        
        self.setNeedsDisplay()
        var t = CGAffineTransform.identity
        let distance = CGFloat(sizeCenter.getDistance(point: location))
        let scale =  distance / initialDistance
        t = t.rotated(by: -angleDiff)
        t = t.scaledBy(x:  CGFloat(scale), y: CGFloat(scale))
       
        self.transform  =  t
        
        self.setNeedsDisplay()
         print("\(center )    \(scale)")
    }
   
}

Update Here's my getDistance Method which is written in CGPoint's extension

func getDistance(point: CGPoint) -> CGFloat {
        let fx = (point.x - self.x);
        let fy = (point.y - self.y);
        
        return sqrt((fx*fx + fy*fy))
    }

Looks like the angle is working as expected, but not the scaling. I can see that each time you begin a drag, you take account of the current angle of the view, but you don't take account of the current scaling. So the answer lies in scaling the view relative to its prior state each time you begin the drag.

I can't test this without a running example but I think that if you make this small change you may get the desired result:

if recognizer.state == .began {
    initialDistance = sizeCenter.getDistance(point: location) / sqrt(transform.a * transform.a + transform.c * transform.c)